{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Offline Estimates of Online Metrics using Causal Inference\n",
    "### Author: aarshay jain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction / Motivation\n",
    "\n",
    "If an offline model has a better AUC (or any other ML metric) than its production counterpart, is it necessarily better for business? Does a higher AUC imply improvement in business metrics? How much improvement in AUC is good enough to justify deployment of a new model or running an A/B test?\n",
    "\n",
    "These are some of the common questions which torment ML practitioners in their daily jobs, especially when building user-facing ML products. Some common scenarios behind these concerns are:\n",
    "- ML applications are trying to drive a product which is judged by business metrics like click rates, revenue, engagement, etc. These are dependent on user feedback/interaction in some form, which is hard to estimate offline.\n",
    "- ML models are typically deployed along with some business heuristics which control how the model output translates to product action, such as which song to play or which ad to show.\n",
    "- Many applications chain predictions from multiple models to take an action. For example, the decision of which ad to show could depend on ML models for click rate predictions and demand forecasts along with business constraints like inventory and user eligibility.\n",
    "\n",
    "In such scenarios, typical ML metrics like accuracy, AUC-ROC, precision-recall, etc. for individual models are generally not enough to tell whether an offline model is a significant improvement over the model in production. A/B tests are typically used for such evaluation, but they are expensive to run both in terms of cost and time. Counterfactual evaluation techniques, inspired from causal inference literature, provide a methodology of using production logs to estimate online metrics like click rate, revenue, etc. Once we have reliable offline estimates, they act act as great first filters on offline models to help decide which ones to A/B test in production."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advertising Example\n",
    "\n",
    "Note: The example and following mathematical formulation used here is a sligtly toned-down version of the example used in [1]. This is not original research, but an attempt to summarize the idea with intuition along with application on simulated data.\n",
    "\n",
    "Let's take an example from the advertising world to understand this better. Consider two sides of the following scenario:\n",
    "- User side - user visits a website and receives an ad; if the user likes the ad, they click on the ad, otherwise they do not\n",
    "- Business side - an ML system receives a request with current user context and picks an ad to display\n",
    "\n",
    "This system can be defined using the following variables:\n",
    "- user-intent (u): A user visits the website with some intent. Eg, user visits amazon.com to buy a shoe\n",
    "- user-context (x): User starts browsing on the website and their activity is packaged as a context vector\n",
    "- ad inventory (c): Inventory of all ads available to show\n",
    "- bids (b): A system which bids a price per click for ads from the ad inventory\n",
    "- selected_ad (a): A final ad selected based on the bids and click estimates\n",
    "- user action (y): Binary 1 if user clicks on the shown ad and 0 otherwise\n",
    "- revenue (z): Some form of revenue ($$) generated from the interaction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Causal Graph\n",
    "The system with the above variables can be depicted as a causal graph as following:\n",
    "\n",
    "<img src=\"fig1-causal_graph.png\">\n",
    "\n",
    "From this graph, we can observe the dependence/relationship between different variables:\n",
    "- u,v are independent variables, also known as exogenous variables\n",
    "- x = f(u)\n",
    "- b = f(x,v)\n",
    "- a = f(x,b)\n",
    "- y = f(a,u)\n",
    "- z = f(y,b)\n",
    "\n",
    "With this understanding, we can model the join probability of the entire system as a probabilistic generative model as:\n",
    "\n",
    "$$P(w) = P(u,v).P(x|u).P(b|x,v).P(a|x,b).P(y|u,a).P(z|y)$$\n",
    "where w is the set of all variables.\n",
    "\n",
    "Intuitively, we started with independent variables and then chained more variables together following the graph, while conditioning on the variables known at the time of determining a new variable. Note that this is an acyclic graph, i.e. if a causes b, then there is no causal path from b to a."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling Interventions\n",
    "\n",
    "In this system, let's say click rate is the business metric we're trying to maximize. Click rate is defined as the proportion of advertisements clicked by users, across user sessions. Suppose we have a production system running and now we develop a new model for selecting an ad, i.e. we have a new way of implementing variable `a`. We want to estimate the click rate of this model as compared to the production model. \n",
    "\n",
    "This system of equations allows us to model interventions as algebraic manipulations i.e. we can change some interim distributions and model a different output for a given input. Refer to Pearl's do-calculus for more details."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Isolation Assumption\n",
    "\n",
    "Before moving forward, let's understand one of the core assumptions of this model: like any causal graph, this graph assumes that the exogenous variables don't have any backdoor path into the network, i.e. there are no common cause between the exogenous variables (u,v) and other variables in the network. For example, suppose an external cause (e) exists which modifies the causal graph as:\n",
    "\n",
    "<img src=\"fig2-backdoor-path.png\">\n",
    "\n",
    "In this case, the causal path represented in red are backdoor paths which invalidate the previously-defined system of equations. Another way of stating this assumption is that saying we assume all observations of exogenous variables are independentally sampled from an unknown but fixed joint distribution. This is the isolation assumption. Most causal grahs make this assumption, and we might be able to argue practical backdoor paths which are not accounted for because such causes cannot be measured/modeled. It is important to be mindful of this assumption while analyzing results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Counterfactual Analysis\n",
    "Now let's dive in to counterfactual analysis principles and try to answer the question posed above.\n",
    "\n",
    "## What is Counterfactual?\n",
    "\n",
    "The question \"what would happen if we replace a current model M with a new model M'\" is counterfactual in the sense that we are not actually making the change and impacting user experience. We are just trying to estimate business metrics in the scenario if we were to hypothetically deploy the model M'. \n",
    "\n",
    "## Analogy to Traditional ML\n",
    "\n",
    "Let's try to compare this scenario with a traditional supervised learning scenario. While training models in a supervised learning setting, we use some independent variables x and true labels y, then we try to estimate y as y' = f(x). y' is kind of a counterfactual estimate of what would happen if, instead of the system generating the data, the model f(x) was used. Then we define a loss function and optimize the model. All of this works because f(x) is fully defined, which is not the case in our problem (i.e. there is no way to know how a user would have interacted if a different ad would be shown). Thus, we need some work-around so that we can estimate the metric without fully defining each component of the system."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markov Factor Replacement\n",
    "\n",
    "Now let's try to perform algebraic manipulations in our system of equations. Say we have a new model M' for selection of an ad given bids. This will impact just one component of the equation:\n",
    "\n",
    "$$P(a|x,b) -> P'(a|x,b)$$\n",
    "The join distribution of the resulting system will be:\n",
    "$$P'(w) = P(u,v).P(x|u).P(b|x,v).P'(a|x,b).P(y|u,a).P(z|y)$$\n",
    "Note that only the one distribution has changed\n",
    "\n",
    "The click rate of the system can be defined as the expectation of click per impression:\n",
    "\n",
    "$$r = \\int_w yP(w)$$\n",
    "Intuitively, this can be understood as an average of the clicks occuring in different context-action scenarios, represented by w, weighted by the probability distribution of w, which is a function of the user activity and production model M.\n",
    "\n",
    "For a new model M', the click rate would be:\n",
    "\n",
    "$$r' = \\int_{w} yP'(w)$$\n",
    "To determine click rate for a new model M', we can simply adjust the probability distribution using the new model M' given the same inputs from user activity.\n",
    "\n",
    "This can be rewritten as:\n",
    "\n",
    "$$r' = \\int_{w} y\\frac{P'(w)}{P(w)}P(w)$$\n",
    "$$r' = \\int_{w} y\\frac{P'(a|x,b)}{P(a|x,b)}P(w)$$\n",
    "Assuming $P(a|x,b)>0$ throughout the domain of w\n",
    "\n",
    "Using the law of large numbers, we can approximate $r'$ as:\n",
    "$$r' \\approx \\frac{1}{n} \\sum_{i=1}^{n} y_i \\frac{P'(a|x,b)}{P(a|x,b)}$$\n",
    "\n",
    "Observe how we have eliminated most of the components of the system from the final estimate. This is super powerful, because now we don't need to fully define the joint distribution $P'(w)$ for the new model M', but only the parts impacted by the intervention/change, which is typically easy to determine as the intervention is controlled.\n",
    "\n",
    "This idea can be generalized for any given metric $l(w)$, the counterfactual estimate of an offline model M' giving a probability distribution $P'(w)$ can be determined using log data with probability distribution $P(w)$ as:\n",
    "$$r' \\approx \\frac{1}{n} \\sum_{i=1}^{n} l(w_i) \\frac{P'(w)}{P(w)}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Intuition\n",
    "Lets try to understand this concept with a small example. Suppose there are 5 data points:\n",
    "\n",
    "| context | p(M) | y | p(M') \n",
    "|------|------|------|------|\n",
    "| w1 | 0.8 | 1 | 0.3 |\n",
    "| w2 | 0.6 | 1 | 0.5 |\n",
    "| w3 | 0.2 | 0 | 0.4 |\n",
    "| w4 | 0.1 | 0 | 0.3 |\n",
    "| w5 | 0.7 | 1 | 0.6 |\n",
    "\n",
    "Here each row is a context where an ad was shown. p(M) is the logged probability of showing the ad by the production model M and p(M') is the probaiblity that an offline model M' will show the same ad. y is the user action, 1 if clicked 0 if not. We can observe that the production model usually has a high probability of showing when the user clicked, hence it's a better model.\n",
    "\n",
    "The estimator will give a similar inference. Whenever there is a show, i.e. $l(w_i)=1$, $P(w)>P'(w)$ hence $r>r'$. Thus, we are able to get back our intuitive ordering of models from the counterfactual estimate.\n",
    "\n",
    "## Constraints / Practical Considerations\n",
    "If we carefully observe the final equation, it puts a constraint on the model being evaluated: the model has to be probabilistic in nature, and we can determine the probability of taking the exact same action which the logging model took, which may not always be trivial. Suppose that we only have the final action a' taken by the new model for each log entry. Then we can also re-write the same equation by taking inspiration from propensity score matching based approaches [2]:\n",
    "\n",
    "$$r' \\approx \\frac{1}{n} \\sum_{i=1}^{n} \\frac{l(w_i)\\ \\mathbb{1}(a' == a) }{P(w)}$$\n",
    "\n",
    "Intuitively, this can be understood as considering the new model to be probabilistic and making $P'(w) = 0$ wherever the new model takes a different action than the logged model. \n",
    "\n",
    "In practice, using the matching approach works if we have a few number of actions and we expect to match a fair number of actions from the logged dataset. However, there are numerous scenarios like recommendation, information retrieval, or multi-arm bandits where we may not have sufficient data to get enough matches for reliable estimates resulting in high variance. Please note that for the sake of simplicity, we will be not be discussing variance considerations in this article, but feel free to read more in [1] [2]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Simulation \n",
    "\n",
    "At this point, hopefully you have developed some some intuitive and mathematical understanding of counterfactual analysis. Let's take it a step further using a similuated example, similar to the one we've been working with. Suppose:\n",
    "- We have 3 ads in inventory, valid for all users\n",
    "- We simulate N number of user contexts, i.e. N different user scenarios, each with an independent click probability on one of the ads\n",
    "- We collect some online data by serving ads randomly to users in different contexts and observing click behavior. Random serving is a good way to collect unbiased online data to evaluate online models and should be adopted when possible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pylab as plt\n",
    "from uuid import uuid4\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data Preparation\n",
    "Let's simulate some logged data which is similar to something coming from a production system.\n",
    "\n",
    "### User Contexts\n",
    "Let's start by defining 10,000 user context (x) ids and define a probability distribution on their occurence, i.e. some contexts are more likely to repeat than others."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set user contexts\n",
    "num_contexts = 10000\n",
    "user_contexts = np.asarray([\"context_{}\".format(i) for i in range(num_contexts)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD4CAYAAADrRI2NAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAT8klEQVR4nO3de5BkVX3A8e8viPiMgAy6LosLulEhJWCmVpSqFKgJDxNXK2rABBfFrKWQaLRSolYClYoVUpVINEasFYiQUpGgFhuDRkQMMQRxQEQeEpdlA+uusDyVGDGLv/xxz7B3e3tmeqYf8zjfT9VU3z59H785c/vXp885905kJpKkevzSfAcgSRotE78kVcbEL0mVMfFLUmVM/JJUmSfMdwAA++23X65cuXK+w5CkReX666+/LzPHZrvdgkj8K1euZGJiYr7DkKRFJSL+ey7b2dUjSZUx8UtSZUz8klQZE78kVcbEL0mVMfFLUmVM/JJUGRO/JFXGxC9JlVkQV+5K82HlGf+yy/PNZ796niKRRssWvyRVxsQvSZUx8UtSZezjV1U6+/WlGtnil6TKmPglqTImfkmqjIlfkirj4K40R+2BYi/+0mJii1+SKmPil6TKmPglqTImfkmqjIO7UhcO3GopM/FrSfLWDNLU7OqRpMrM2OKPiCcBVwN7lfUvzcwzI+Ig4GJgX+AG4OTM/HlE7AVcBPwacD/wu5m5eUjxSwMz1bcEu3201PTS4n8UeEVmHgYcDhwXEUcCfwWck5mrgAeBU8v6pwIPZubzgXPKepKkBWLGFn9mJvBIebpn+UngFcCbSvmFwFnAucCasgxwKfCxiIiyH2lRc+xAS0FPffwRsUdE3AjcC1wB3AE8lJk7yipbgOVleTlwN0B5/WHgmV32uS4iJiJiYvv27f39FpKknvU0qyczHwMOj4i9gS8CL+q2WnmMaV5r73M9sB5gfHzcbwNaMhwT0EI3q1k9mfkQ8A3gSGDviJj84DgA2FqWtwArAMrrzwAeGESwkqT+zZj4I2KstPSJiCcDrwJuA64CXl9WWwtcVpY3lOeU179u/74kLRy9dPUsAy6MiD1oPiguycwvRcStwMUR8RfAd4Dzy/rnA/8YERtpWvonDiFuSdIc9TKr5ybgiC7lm4DVXcp/BrxhINFJkgbOK3clqTLeq0caAOf3azGxxS9JlTHxS1JlTPySVBkTvyRVxsQvSZVxVo80RN63RwuRLX5JqoyJX5IqY1ePlgwvopJ6Y4tfkipj4pekypj4Jaky9vFL88BpnppPtvglqTImfkmqjF090jyz20ejZuKXRsTrDLRQmPi1qJlMpdmzj1+SKmPil6TKzJj4I2JFRFwVEbdFxC0R8a5SflZE/DAibiw/J7S2eX9EbIyI2yPi2GH+ApKk2emlj38H8N7MvCEing5cHxFXlNfOycy/bq8cEYcAJwKHAs8BvhYRv5KZjw0ycEnS3MyY+DNzG7CtLP8kIm4Dlk+zyRrg4sx8FLgzIjYCq4H/HEC8kgO6Up9m1ccfESuBI4BvlaLTI+KmiLggIvYpZcuBu1ubbaHLB0VErIuIiYiY2L59+6wDlyTNTc+JPyKeBnweeHdm/hg4F3gecDjNN4K/mVy1y+a5W0Hm+swcz8zxsbGxWQcuSZqbnhJ/ROxJk/Q/nZlfAMjMezLzscz8BfBJmu4caFr4K1qbHwBsHVzIkqR+9DKrJ4Dzgdsy88Ot8mWt1V4H3FyWNwAnRsReEXEQsAq4bnAhS5L60cusnqOAk4HvRcSNpewDwEkRcThNN85m4O0AmXlLRFwC3EozI+g0Z/RI0sLRy6yeb9K93/7yabb5EPChPuKSJA2JV+5KUmVM/JJUGRO/JFXGxC9JlfF+/FoUarlNg/+NS6Ngi1+SKmOLX1qgbP1rWGzxS1JlTPySVBkTvyRVxsQvSZVxcFcLVi1TOKVRs8UvSZUx8UtSZUz8klQZE78kVcbEL0mVMfFLUmVM/JJUGRO/JFXGxC9JlTHxS1JlZkz8EbEiIq6KiNsi4paIeFcp3zciroiIH5THfUp5RMRHI2JjRNwUES8Z9i8hSepdLy3+HcB7M/NFwJHAaRFxCHAGcGVmrgKuLM8BjgdWlZ91wLkDj1qSNGczJv7M3JaZN5TlnwC3AcuBNcCFZbULgdeW5TXARdm4Ftg7IpYNPHJJ0pzMqo8/IlYCRwDfAp6Vmdug+XAA9i+rLQfubm22pZR17mtdRExExMT27dtnH7kkaU56TvwR8TTg88C7M/PH063apSx3K8hcn5njmTk+NjbWaxiSpD71lPgjYk+apP/pzPxCKb5nsgunPN5byrcAK1qbHwBsHUy4kqR+9TKrJ4Dzgdsy88OtlzYAa8vyWuCyVvmby+yeI4GHJ7uEJEnzr5f/wHUUcDLwvYi4sZR9ADgbuCQiTgXuAt5QXrscOAHYCPwUeMtAI5Yq1P5vZJvPfvU8RqKlYMbEn5nfpHu/PcAru6yfwGl9xiVJGhL/564WFP/PrjR83rJBkipji1/zzla+NFomfmmRcaBX/bKrR5IqY+KXpMqY+CWpMvbxS4uY/f2aCxO/5oUzeaT5Y1ePJFXGxC9JlTHxS1Jl7OPXyNivLy0MtvglqTImfkmqjIlfkipj4pekypj4JakyJn5JqozTOTVUTuGUFh5b/JJUGVv80hLhnTrVqxlb/BFxQUTcGxE3t8rOiogfRsSN5eeE1mvvj4iNEXF7RBw7rMAlSXPTS1fPp4DjupSfk5mHl5/LASLiEOBE4NCyzccjYo9BBStJ6t+MiT8zrwYe6HF/a4CLM/PRzLwT2Ais7iM+SdKA9TO4e3pE3FS6gvYpZcuBu1vrbCllu4mIdRExERET27dv7yMMSdJszDXxnws8Dzgc2Ab8TSmPLutmtx1k5vrMHM/M8bGxsTmGIUmarTkl/sy8JzMfy8xfAJ9kZ3fOFmBFa9UDgK39hShJGqQ5TeeMiGWZua08fR0wOeNnA/CZiPgw8BxgFXBd31FqUfGiLWlhmzHxR8RngaOB/SJiC3AmcHREHE7TjbMZeDtAZt4SEZcAtwI7gNMy87HhhC5JmovI7NoFP1Lj4+M5MTEx32FoQGzxLyxezLV0RcT1mTk+2+28ZYMkVcbEL0mV8V49Ggi7d6TFwxa/JFXGxC9JlTHxS1JlTPySVBkHd6WK+M9aBLb4Jak6Jn5JqoyJX5IqY+KXpMqY+CWpMs7qkSrlDJ962eKXpMqY+CWpMiZ+SaqMffzSEucts9XJFr8kVcbEL0mVMfFLUmVM/JJUmRkHdyPiAuC3gHsz81dL2b7A54CVwGbgjZn5YEQE8BHgBOCnwCmZecNwQtd8c9BQWpx6afF/Cjiuo+wM4MrMXAVcWZ4DHA+sKj/rgHMHE6YkaVBmbPFn5tURsbKjeA1wdFm+EPgG8L5SflFmJnBtROwdEcsyc9ugApY0eN6+oS5z7eN/1mQyL4/7l/LlwN2t9baUst1ExLqImIiIie3bt88xDEnSbA16cDe6lGW3FTNzfWaOZ+b42NjYgMOQJE1lron/nohYBlAe7y3lW4AVrfUOALbOPTxJ0qDNNfFvANaW5bXAZa3yN0fjSOBh+/claWHpZTrnZ2kGcveLiC3AmcDZwCURcSpwF/CGsvrlNFM5N9JM53zLEGLWPHIKp7T49TKr56QpXnpll3UTOK3foCRJw+PdOdWV0/ukpctbNkhSZUz8klQZu3o0Iwd0paXFFr8kVcbEL0mVMfFLUmVM/JJUGRO/JFXGxC9JlTHxS1JlnMcvaRed1214y46lxxa/JFXGFr+knnnzvqXBFr8kVcYWf+VswUn1MfHrcd6MTaqDXT2SVBlb/JKm5TfBpcfEL2lOHB9avOzqkaTKmPglqTJ9dfVExGbgJ8BjwI7MHI+IfYHPASuBzcAbM/PB/sKUJA3KIPr4j8nM+1rPzwCuzMyzI+KM8vx9AziOBsTBOqluw+jqWQNcWJYvBF47hGNIkuao38SfwFcj4vqIWFfKnpWZ2wDK4/59HkOSNED9dvUclZlbI2J/4IqI+H6vG5YPinUABx54YJ9hSJJ61VeLPzO3lsd7gS8Cq4F7ImIZQHm8d4pt12fmeGaOj42N9ROGJGkW5pz4I+KpEfH0yWXgN4GbgQ3A2rLaWuCyfoOUJA1OP109zwK+GBGT+/lMZn4lIr4NXBIRpwJ3AW/oP0xJC5lX8S4uc078mbkJOKxL+f3AK/sJSpI0PF65K0mVMfFLUmVM/JJUGW/LXAFv0aCFwAHghcPEv0SZ7LUQeB4uTHb1SFJlTPySVBkTvyRVxsQvSZUx8UtSZSIz5zsGxsfHc2JiYr7DWPScQaHFyKmdcxcR12fm+Gy3czrnImSCl9QPu3okqTK2+CXNK6/oHT1b/JJUGVv8khYkvwkMj4l/kXBAV9Kg2NUjSZVxHv8CZitf2p3dPjvNdR6/LX5Jqox9/JIWrakGgB0Ynp6Jf4R6OUklTc/3S/9M/PPEk1caLN9TvRta4o+I44CPAHsA52Xm2cM61kLgV0tpYerlA6G29+xQEn9E7AH8PfAbwBbg2xGxITNvHcbxFho/BCQtZMNq8a8GNmbmJoCIuBhYAww88febZPvpd+/leH79lBa+ft/vsx1knu/G4VDm8UfE64HjMvNt5fnJwEsz8/TWOuuAdeXpC4Db53i4/YD7+gh3WIxrdoxrdoxrdpZqXM/NzLHZbjSsFn90KdvlEyYz1wPr+z5QxMRcLmAYNuOaHeOaHeOaHePa1bAu4NoCrGg9PwDYOqRjSZJmYViJ/9vAqog4KCKeCJwIbBjSsSRJszCUrp7M3BERpwP/SjOd84LMvGUYx2IA3UVDYlyzY1yzY1yzY1wtC+ImbZKk0fEmbZJUGRO/JNUmM0f2AxxHM19/I3BGl9f3Aj5XXv8WsLL12vtL+e3AsTPtEzio7OMHZZ9PnOIYa1vbf6XzGCOK6T00F7fdBFwJPLe1j6SZEXUjsGHEdXUKsL0c+0bgba193EMz//gHwNoRx3VOK6b/Ah4acX2dXsoS2K9VHsBHy2s3AS8ZcX1NFdfvlXhuAq4BDmvt4/+AH5X6mhhxXEcDD7f+ln/W2sdWmnOvc1+jiOtPWjHdDDwGvH6E9fXpUn4zcAGw51TnV2ubtTTn1i7n15S5eIRJfw/gDuBg4InAd4FDOtZ5J/CJsnwi8LmyfEhZfy+aZHBH2d+U+wQuAU4sy58A3tHlGCcBj5TtDwP+tzy2jzGKmI4BnlKW31HWm9zHI/NYV6cAH+vyNzwC2FROzCPL8j6jiqujHv4Q+IcR19cRwEpgM7smjBOAL9O8QY+kSQCjrK+p4np563jHt+I6uKx78zzV19HAl7qcX88vj7fRvB/b+xp6XB318NvA10dcXyfQnEMBfJad78fdzq9Svi/NObUvsA+t82uqn1F29Tx+G4fM/DkweRuHtjXAhWX5UuCVERGl/OLMfDQz76T5xFs91T7LNq8o+6Ds87VdjnE3TaXfSVOp/wac0HGMoceUmVdl5k9L+bXACyf3Ucrmq646rS77eyFwBU3L5JiyfNw8xXUS8J1R1RdAZn4nMzd3iWUNcFE2rgWeDdw1ivqaLq7MvCYzHyxPr6VJNu36+uI81VenyfNrrDxeRPO+bP89Rx3XSTQflKOsr8vLOZTAdTTXQU0eo31+7R0Ry4BjgSsy84Hyd26fX12NMvEvp0m0k7aUsq7rZOYOmq+Bz5xm26nKnwk8VPbReaz2Ns8GftY6xp2t9XY7xhBjajsVuKW1jycBfwC8KyLaiW9Ucf1ORNwUEZcCLy77mNzX5Lpd63fY9RURz6VpRf1ohPU1nc5tHi4/o6ivXp1K07qc3EcCJwN/XG6jstvvMuS4XhYR342ILwMvZXTn14wi4ins7I4ZeX1FxJ7lWF/pPEa/v+MoE/+Mt3GYZp1BlXceI1qvRcd6gz72jL9/RPw+MM6uF7sdCJxJc03E30bE8zpiH2Zc/0zTV/li4Gs0faKdx86Ox5HVF81X6ks7yoddX9Pp3Ca67GtY9TWjiDiGJvF/ulV8FPCnNOfcaRHx6yOM6waae80cBvwd8L4ux563+qLp5vkP4H9aZaOsr48DV2fmv8/xGFMaZeLv5TYOj68TEU8AngE8MM22U5XfR/M16Akd5Z1xbKNpIU4e46DWersdY4gxERGvAj4IvIamH3EFQGZuLet+H/gGTb/kSOoqM+/PzEdL+SdL/axo7Wty3a71O8z6Kk6k6QN9fB8jqK/pdG7zy8DejKa+phURLwbOo+ku+D6719cdNF0Yq0cVV2b+ODMfKcuXl+KDWQD1VUx3fg21viLiTJour/e01hnc7zjdAMAgf2iuEt5EkzwmBzMO7VjnNHYdILmkLB/KrgMkm2gGR6bcJ/BP7Dow+M4ux3gTzWDgQewc3D284xijiOkImhNpVUddHQY8rezjKJoR+0NGWFfLWn+b19H0D28qdXQnzSDXy8ryvqOKqzx/Ac0HZIy6vloxbGbXwcpXs+vg23WjrK9p4jqQpm/55R3n1yE0A4LfpfmmeQ3NXXVHFdez2XkR6Wqa8ZBNNIO7m9h1cPfQUcVVyiYT91NHXV80s+euAZ7cEdNu51fuHNy9k2Zgdx9a59eU+XhUib8EeALN9Ls7gA+Wsj8HXlOWn0TzZt9I86Y5uLXtB8t2twPHT7fPUn5w2cfGss+9pjjGW1vbf7U83gecNcKYvkYz3W9yCtmGso+7gEdpvpl8r5SPsq7+kma84bvAVTSDlJP7uBe4v2xz2SjjKq+dBZzdsY9R1dcf0bSydtC0rM4r5UHzD4juKMcfH3F9TRXXecCD7Dy/Jso+7mzV1y00s1dGGdfp7Dy/rqWZfTS5j20078M7Rh1Xee0UmkHZ9j5GVV87Stnj01ynOr9a27y1HHsj8JaZcrG3bJCkynjlriRVxsQvSZUx8UtSZUz8klQZE78kVcbEL0mVMfFLUmX+H1T/ArLrTXDUAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# assign selection prior to these contexts\n",
    "def random_normal_sample_sum_to_1(size):\n",
    "    sample = np.random.normal(0, 1, size)\n",
    "    sample_adjusted = sample - sample.min()\n",
    "    return sample_adjusted / sample_adjusted.sum()\n",
    "user_context_selection_prior = random_normal_sample_sum_to_1(num_contexts)\n",
    "plt.hist(user_context_selection_prior, bins=100)\n",
    "assert user_context_selection_prior.sum().round(2) == 1.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Click Rate Per Context\n",
    "Let's define 3 ads in our inventory. For the sake of simplicity, let's say the click rate for a given ad in a given context can be one of:\n",
    "- Low: 10%\n",
    "- Medium: 40%\n",
    "- High: 60%\n",
    "\n",
    "Then we can randomly assign which of the ads is low/med/high in which context. The idea is that a good model will pick the high-interaction ad in a given context more often than low-interaction ad."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set ads\n",
    "num_ads = 3\n",
    "ads = np.asarray([\"ad_{}\".format(i) for i in range(num_ads)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assign random priors to contexts\n",
    "ad_interaction_priors = np.asarray([0.1, 0.3, 0.6])\n",
    "user_context_priors = {context:np.random.permutation(ad_interaction_priors) for context in user_contexts}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Random Data Collection\n",
    "Now let's simulate 100,000 iterations where one of the 10,000 user contexts is given as input to the model, a random ad is served, and based on the click prior for that ad in that context an action of click or no click is sampled. The idea here is to generate data similar to production logs of any model, the difference being that we are simulating the user side.\n",
    "\n",
    "The simulated data will contain four columns:\n",
    "1. log_id: represents each logged row\n",
    "2. context_id: represents the user context from our list of 10,000 contexts\n",
    "3. selected_ad: the ad shown by the production model\n",
    "4. user_interaction: binary 1 if the user interacted and 0 otherwise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_iterations = 100000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create empty df for storing logs\n",
    "df_random_serving = pd.DataFrame(\n",
    "    columns = [\"log_id\", \"context_id\", \"selected_ad\", \"user_interaction\"]\n",
    ")\n",
    "\n",
    "# create unique ID for each log entry\n",
    "df_random_serving[\"log_id\"] = [uuid4() for _ in range(num_iterations)]\n",
    "\n",
    "# assign a context id to each log entry\n",
    "df_random_serving[\"context_id\"] = np.random.choice(user_contexts, size=num_iterations, replace=True, p=user_context_selection_prior)\n",
    "\n",
    "# randomly sample an ad to show in that context\n",
    "df_random_serving[\"selected_ad\"] = np.random.choice(ads, size=num_iterations, replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for each log entry, sample an action or click or not using the click probability assigned to the context-ad pair in step 1\n",
    "def sample_action_for_ad(context_id, ad_id):\n",
    "    prior = user_context_priors.get(context_id)[np.where(ads == ad_id)[0][0]]\n",
    "    return np.random.binomial(1, prior)\n",
    "df_random_serving[\"user_interaction\"] = df_random_serving.apply(lambda x: sample_action_for_ad(x[\"context_id\"], x[\"selected_ad\"]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>log_id</th>\n",
       "      <th>context_id</th>\n",
       "      <th>selected_ad</th>\n",
       "      <th>user_interaction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>58898</td>\n",
       "      <td>1394c11d-0fb1-4517-8f04-0a6db33cb382</td>\n",
       "      <td>context_4667</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>83132</td>\n",
       "      <td>4d755f03-3491-4a56-bf49-c9e4876858c6</td>\n",
       "      <td>context_1200</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>19167</td>\n",
       "      <td>d60c7cc1-3a3a-4dd3-83eb-e328e4962ee0</td>\n",
       "      <td>context_1080</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>77984</td>\n",
       "      <td>08ea91f4-7adc-4938-b45a-394a8680103e</td>\n",
       "      <td>context_2656</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>95146</td>\n",
       "      <td>f1b3b8ec-f9a3-42a2-afdd-a5ab69868b3a</td>\n",
       "      <td>context_7795</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>43339</td>\n",
       "      <td>dfe2997b-7e43-4a75-b799-d1027a388a90</td>\n",
       "      <td>context_7035</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20231</td>\n",
       "      <td>59c17bb1-3017-4f0b-b9d2-5c80d725d4f6</td>\n",
       "      <td>context_4727</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>26362</td>\n",
       "      <td>d44ab4bc-297f-4b1c-9a8c-7194f0fd9eab</td>\n",
       "      <td>context_4785</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>72270</td>\n",
       "      <td>1e62f203-f1fb-4003-9899-818e652caa56</td>\n",
       "      <td>context_2784</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>20171</td>\n",
       "      <td>03ff9308-ab23-46cf-aa06-60920c0e85b2</td>\n",
       "      <td>context_6105</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     log_id    context_id selected_ad  \\\n",
       "58898  1394c11d-0fb1-4517-8f04-0a6db33cb382  context_4667        ad_1   \n",
       "83132  4d755f03-3491-4a56-bf49-c9e4876858c6  context_1200        ad_0   \n",
       "19167  d60c7cc1-3a3a-4dd3-83eb-e328e4962ee0  context_1080        ad_2   \n",
       "77984  08ea91f4-7adc-4938-b45a-394a8680103e  context_2656        ad_2   \n",
       "95146  f1b3b8ec-f9a3-42a2-afdd-a5ab69868b3a  context_7795        ad_1   \n",
       "43339  dfe2997b-7e43-4a75-b799-d1027a388a90  context_7035        ad_2   \n",
       "20231  59c17bb1-3017-4f0b-b9d2-5c80d725d4f6  context_4727        ad_1   \n",
       "26362  d44ab4bc-297f-4b1c-9a8c-7194f0fd9eab  context_4785        ad_0   \n",
       "72270  1e62f203-f1fb-4003-9899-818e652caa56  context_2784        ad_0   \n",
       "20171  03ff9308-ab23-46cf-aa06-60920c0e85b2  context_6105        ad_0   \n",
       "\n",
       "       user_interaction  \n",
       "58898                 0  \n",
       "83132                 0  \n",
       "19167                 0  \n",
       "77984                 0  \n",
       "95146                 1  \n",
       "43339                 0  \n",
       "20231                 0  \n",
       "26362                 0  \n",
       "72270                 0  \n",
       "20171                 0  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# a snapshot of the data\n",
    "df_random_serving.sample(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To recap, each row in this simulated log data represents an instance where:\n",
    "- a user visited a website with some intent and generated a user context (x) denoted by context_id.\n",
    "- a production model (random in this case) selected an ad to show to the user\n",
    "- the user interaction was observed and logged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. New Model Estimation\n",
    "\n",
    "### Define Models\n",
    "\n",
    "Now let's define some models for which we can establish an intuition of order of performance on click rate. This will allow us to simulate offline model predictions, use counterfactual estimation to get metrics, and compare them with expected results. \n",
    "\n",
    "One way to do so is to set priors on the likelihood of picking the low/med/high ad using a vector \\[p_low, p_med, p_high\\] for a given context. We can intuitively say that the models with a higher likelihood of picking more performant ads for a context will work better. Note that this is not how an actual model would work because we wouldn't know the click rate prior; think of these as models which have learned these priors with different levels of accuracy.\n",
    "\n",
    "Here's a group of 10 models with increasing expected performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_model_priors = np.atleast_2d([\n",
    "    [0.8, 0.1, 0.1],\n",
    "    [0.7, 0.2, 0.1],\n",
    "    [0.6, 0.2, 0.2],\n",
    "    [0.5, 0.3, 0.2],\n",
    "    [0.5, 0.2, 0.3],\n",
    "    [0.4, 0.3, 0.3],\n",
    "    [0.4, 0.2, 0.4],\n",
    "    [0.3, 0.3, 0.4],\n",
    "    [0.2, 0.35, 0.45],\n",
    "    [0.2, 0.2, 0.6]\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For clarification, let's see this as a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>model_id</th>\n",
       "      <th>prob_low</th>\n",
       "      <th>prob_med</th>\n",
       "      <th>prob_high</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>model_0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>model_1</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>model_2</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>model_3</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>model_4</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>model_5</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>model_6</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>model_7</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>model_8</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.35</td>\n",
       "      <td>0.45</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>model_9</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  model_id prob_low prob_med prob_high\n",
       "0  model_0      0.8      0.1       0.1\n",
       "1  model_1      0.7      0.2       0.1\n",
       "2  model_2      0.6      0.2       0.2\n",
       "3  model_3      0.5      0.3       0.2\n",
       "4  model_4      0.5      0.2       0.3\n",
       "5  model_5      0.4      0.3       0.3\n",
       "6  model_6      0.4      0.2       0.4\n",
       "7  model_7      0.3      0.3       0.4\n",
       "8  model_8      0.2     0.35      0.45\n",
       "9  model_9      0.2      0.2       0.6"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_model_names = np.asarray([\"model_{}\".format(i) for i in range(new_model_priors.shape[0])])\n",
    "pd.DataFrame(\n",
    "    data=np.hstack([np.atleast_2d(new_model_names).T, new_model_priors]),\n",
    "    columns=[\"model_id\", \"prob_low\", \"prob_med\", \"prob_high\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One nice feature of defining a model in this way is that we can actually calculate the expected click rate of each model. Since a model is a set of priors on picking a low/med/high ad, and we have already fixed the interaction rates of low/med/high ads, we can just take a dot product to estimate the click rate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.17 , 0.19 , 0.24 , 0.26 , 0.29 , 0.31 , 0.34 , 0.36 , 0.395,\n",
       "       0.44 ])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# expected interaction rate:\n",
    "expected_interaction_rates = np.dot(new_model_priors, np.atleast_2d(ad_interaction_priors).T)\n",
    "expected_interaction_rates.ravel()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that expected click rates are in the order that we expect. Now we'll try to sample ad selections using each policy and use the counterfactual technique learned above to see if we can estimate these click rates using only the logged data and sampled outcomes from each model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimate Click Rate: Propensity Matching\n",
    "\n",
    "First, let's use the propensity matching estimate:\n",
    "$$r' \\approx \\frac{1}{n} \\sum_{i=1}^{n} \\frac{y_i\\ \\mathbb{1}(a' == a) }{P(w)}$$\n",
    "where:\n",
    "- $y_i$: user interaction\n",
    "- $a'$: action suggested by new model\n",
    "- $a$: action taking by production model\n",
    "- $P(a|x,b)$: logged probability of the production model (random serving)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the same context ids as logged data:\n",
    "df_new_models_matching = df_random_serving.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_ad_for_context_n_model(context_id, model_priors):\n",
    "    # get ad interaction priors for the given context\n",
    "    interaction_priors = user_context_priors.get(context_id)\n",
    "    \n",
    "    # get the selection prior for the given model based on interaction priors\n",
    "    selection_priors = model_priors[np.argsort(np.argsort(interaction_priors))]\n",
    "    \n",
    "    # select an ad using the priors and log the selection probability\n",
    "    selected_ad = np.random.choice(ads, None, replace=False, p=selection_priors)\n",
    "#     selected_ad_prior = selection_priors[ads.tolist().index(selected_ad)]\n",
    "    return selected_ad\n",
    "\n",
    "for policy_name, model_prior in zip(new_model_names, new_model_priors):\n",
    "    df_new_models_matching.loc[:, policy_name] = df_new_models_matching[\"context_id\"].apply(lambda x: sample_ad_for_context_n_model(x, model_prior))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>log_id</th>\n",
       "      <th>context_id</th>\n",
       "      <th>selected_ad</th>\n",
       "      <th>user_interaction</th>\n",
       "      <th>model_0</th>\n",
       "      <th>model_1</th>\n",
       "      <th>model_2</th>\n",
       "      <th>model_3</th>\n",
       "      <th>model_4</th>\n",
       "      <th>model_5</th>\n",
       "      <th>model_6</th>\n",
       "      <th>model_7</th>\n",
       "      <th>model_8</th>\n",
       "      <th>model_9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>7084</td>\n",
       "      <td>192d9c87-3dc4-4e42-9326-967424fcdd44</td>\n",
       "      <td>context_7614</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>0</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5455</td>\n",
       "      <td>24db9e00-4caf-42f6-b1cf-85bbaee85850</td>\n",
       "      <td>context_5651</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>0</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>67259</td>\n",
       "      <td>79327ad7-a240-4ebe-aa66-9e915272bbcc</td>\n",
       "      <td>context_8476</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>0</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>67292</td>\n",
       "      <td>8d05a652-bdcf-44d3-9fd5-e9aebc6bba60</td>\n",
       "      <td>context_6875</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>92141</td>\n",
       "      <td>81e2c4fe-bdd1-4241-8782-84253731832f</td>\n",
       "      <td>context_2931</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>ad_2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     log_id    context_id selected_ad  \\\n",
       "7084   192d9c87-3dc4-4e42-9326-967424fcdd44  context_7614        ad_2   \n",
       "5455   24db9e00-4caf-42f6-b1cf-85bbaee85850  context_5651        ad_2   \n",
       "67259  79327ad7-a240-4ebe-aa66-9e915272bbcc  context_8476        ad_0   \n",
       "67292  8d05a652-bdcf-44d3-9fd5-e9aebc6bba60  context_6875        ad_1   \n",
       "92141  81e2c4fe-bdd1-4241-8782-84253731832f  context_2931        ad_1   \n",
       "\n",
       "       user_interaction model_0 model_1 model_2 model_3 model_4 model_5  \\\n",
       "7084                  0    ad_2    ad_2    ad_1    ad_2    ad_2    ad_1   \n",
       "5455                  0    ad_1    ad_1    ad_2    ad_0    ad_0    ad_2   \n",
       "67259                 0    ad_2    ad_0    ad_1    ad_2    ad_0    ad_1   \n",
       "67292                 0    ad_2    ad_2    ad_2    ad_0    ad_2    ad_2   \n",
       "92141                 0    ad_0    ad_0    ad_1    ad_1    ad_2    ad_2   \n",
       "\n",
       "      model_6 model_7 model_8 model_9  \n",
       "7084     ad_0    ad_1    ad_1    ad_1  \n",
       "5455     ad_2    ad_1    ad_1    ad_2  \n",
       "67259    ad_0    ad_1    ad_2    ad_2  \n",
       "67292    ad_0    ad_2    ad_0    ad_2  \n",
       "92141    ad_1    ad_2    ad_0    ad_2  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new_models_matching.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that for each log entry we've computed the ad selection from all new models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# match and estimate:\n",
    "estimates_matching = []\n",
    "for i in range(len(new_model_names)):\n",
    "    model = \"model_{}\".format(i)\n",
    "    matching_mask = (df_new_models_matching[\"selected_ad\"] == df_new_models_matching[model].values).astype(int)\n",
    "    \n",
    "    # the logging policy was random so we know P(w) = 1/3\n",
    "    estimate = (df_new_models_matching[\"user_interaction\"] * matching_mask / 0.333).sum() / df_new_models_matching.shape[0]\n",
    "    estimates_matching.append(estimate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAFDCAYAAAAJXRsnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeVyVVeLH8c+57ILgiqCo4C4gKuKeS5qKlVm2mLZvZmbLNE3bTFO/lmmZGtsXp2kvbVHTzD3XTEvIcsHcURFUBBWQ/d7z+wMiMkw09AJ+36/XvOLe5zyX78WS7zzPuecYay0iIiIiUrUc7g4gIiIiUhupZImIiIicBipZIiIiIqeBSpaIiIjIaaCSJSIiInIaeLo7wLEaNWpkw8PD3R1DRERE5IQSExMPWmsbV3SsUiXLGBMPvAh4AG9Za58+zrjLgM+A7tbaBGNMOLAJ2Fw6ZLW1dvwffa/w8HASEhIqE0tERETErYwxu4537IQlyxjjAbwKDAFSgDXGmFnW2qRjxtUF7gS+O+Yltltru5x0ahEREZEarDJzsnoA26y1O6y1hcBUYGQF4x4HngXyqzCfiIiISI1UmZLVDNhT7nFK6XNljDFdgebW2tkVnB9hjFlrjFlmjOlX0TcwxowzxiQYYxLS09Mrm11ERESk2qrMnCxTwXNle/EYYxzAJOD6CsalAS2stRnGmG7AF8aYKGtt1m9ezNrJwGSAuLi43+3zU1RUREpKCvn5ukhWXfn6+hIWFoaXl5e7o4iIiFQLlSlZKUDzco/DgNRyj+sC0cBSYwxACDDLGHORtTYBKACw1iYaY7YD7YCTmtmekpJC3bp1CQ8Pp/R7SDVirSUjI4OUlBQiIiLcHUdERKRaqMztwjVAW2NMhDHGG7gSmPXLQWvtEWttI2ttuLU2HFgNXFT66cLGpRPnMca0AtoCO042ZH5+Pg0bNlTBqqaMMTRs2FBXGkVERMo54ZUsa22xMWYiMJ+SJRzettZuNMY8BiRYa2f9wen9gceMMcWAExhvrc08laAqWNWb/nxERER+q1LrZFlr5wBzjnnun8cZO7Dc19OAaX8in4iIiEiNpG11qrHk5GQ+/vjjkz7v+uuv5/PPPz/l77t06VK+/fbbUz5fREREVLKqtVMtWZVRXFx83GMqWSIiUuM5iyB7v1sjqGRV0ocffkiPHj3o0qULt956K06nk127dtG2bVsOHjyIy+WiX79+LFiwgOTkZDp06MB1111HTEwMl112Gbm5uQAkJiYyYMAAunXrxrBhw0hLSwNg27ZtnHfeeXTu3JnY2Fi2b9/OAw88wIoVK+jSpQuTJk3C6XTyt7/9je7duxMTE8Obb74JlHy6b+LEiURGRnLBBRdw4MCBCt/DwIEDeeihhxgwYAAvvvgiX375JT179qRr166cd9557N+/n+TkZN544w0mTZpEly5dWLFiBenp6Vx66aV0796d7t27s3LlyjPzQxcRETkVRzNwvjeSgndGQHGh22JUuw2iT+T/vtxIUmrWiQeehMimgTwyIuq4xzdt2sQnn3zCypUr8fLyYsKECXz00Udce+213H///YwfP56ePXsSGRnJ0KFDSU5OZvPmzfzvf/+jb9++3Hjjjbz22mvcdddd3HHHHcycOZPGjRvzySef8Pe//523336bq666igceeIBLLrmE/Px8XC4XTz/9NM899xyzZ5es8Tp58mSCgoJYs2YNBQUF9O3bl6FDh7J27Vo2b97M+vXr2b9/P5GRkdx4440VvpfDhw+zbNkyAA4dOsTq1asxxvDWW2/x7LPP8vzzzzN+/HgCAgK49957ARg7dix/+ctfOOecc9i9ezfDhg1j06ZNVfpnICIiUiX2baDww9GQs59nvSfygPHEXSs41riS5Q5ff/01iYmJdO/eHYC8vDyCg4MBuPnmm/nss8944403+PHHH8vOad68OX379gXg6quv5qWXXiI+Pp4NGzYwZMgQAJxOJ6GhoWRnZ7N3714uueQSoGRhz4osWLCAdevWlc23OnLkCFu3bmX58uWMGTMGDw8PmjZtyqBBg477XkaPHl32dUpKCqNHjyYtLY3CwsLjrnG1aNEikpJ+3aoyKyuL7Oxs6tat+8c/OBERkTPIlTQL57RxZBb78nfvJxg/9gq8PNx3067Glaw/uuJ0ulhrue6663jqqad+dyw3N5eUlBQAcnJyyorHsUsaGGOw1hIVFcWqVat+cywrq3JX5qy1vPzyywwbNuw3z8+ZM6fSSyj4+/uXfX3HHXdwzz33cNFFF7F06VIeffTRCs9xuVysWrUKPz+/Sn0PERGRM8paji58Cv9vn2GdqzUfR/yL50cPol4db7fG0pysShg8eDCff/552VynzMxMdu3aBcD999/PVVddxWOPPcYtt9xSds7u3bvLytSUKVM455xzaN++Penp6WXPFxUVsXHjRgIDAwkLC+OLL74AoKCggNzcXOrWrUt2dnbZaw4bNozXX3+doqIiALZs2cLRo0fp378/U6dOxel0kpaWxpIlSyr1vo4cOUKzZiXbUL733ntlzx/7fYcOHcorr7xS9rj8FTsRERG3KjxK+jtj8P/2Gb5w9SNp2FSeuX6Y2wsWqGRVSmRkJE888QRDhw4lJiaGIUOGkJaWxrJly1izZk1Z0fL29uadd94BoGPHjrz33nvExMSQmZnJbbfdhre3N59//jn3338/nTt3pkuXLmWf4vvggw946aWXiImJoU+fPuzbt4+YmBg8PT3p3LkzkyZN4uabbyYyMpLY2Fiio6O59dZbKS4u5pJLLqFt27Z06tSJ2267jQEDBlTqfT366KNcfvnl9OvXj0aNGpU9P2LECGbMmFE28f2ll14iISGBmJgYIiMjeeONN6r+hywiInKSCjN2se+FgTTYNY83fW+g420fM7Zvu2qzQLax9nf7MbtVXFycTUj47daGmzZtomPHjm5KdPKSk5O58MIL2bBhg7ujnFE17c9JRERqrn3rF+M7/XocrkJmtHqM0WNvwtfL44znMMYkWmvjKjpW4+ZkiYiIyNlt7RcvEr32/9hrgtkzbArX9enr7kgVUsk6DcLDw8+6q1giIiKn29G8fH747230y5zOjz6xBN/4Mf1CQt0d67hUskRERKTa27R9F7kfX00/5zoSm46l8w0v4unl/sntf0QlS0RERKotay3T5i2i++rbaW0y2N7nWboNvdXdsSpFJUtERESqpYM5BXz43pvcdOBfuDx9yR89i9btquf8q4qoZImIiEi1882WdNZOfZQ7nR9xKKgjDW76DBMU5u5YJ0XrZJ0GS5cuLVv/6lQFBAT8qfNfeOGFsk2pRUREaooip4vnvvqRgx9cyx2uD8lpO4KGE7+ucQULVLJOi6ooWSdircXlch33uEqWiIjUNLszcrn11S8Z+t31XOSxiqKBDxN41fvgXcfd0U6JSlYlXXzxxXTr1o2oqCgmT55c9vy8efOIjY2lc+fODB48mOTkZN544w0mTZpUtmL69ddfX7apM/x6lSonJ4fBgwcTGxtLp06dmDlz5h9mSE5OpmPHjkyYMIHY2Fj27NnDbbfdRlxcHFFRUTzyyCMAvPTSS6SmpnLuuedy7rnnAiWbS/fu3ZvY2Fguv/xycnJyqvpHJCIicspm/ZTKgy/9j2cy7yTSez+OKz/Ga+C9UE1Wbz8VNW/F97kPwL71VftNQzrB8Kf/cEhmZiYNGjQgLy+P7t27s2zZMlwuF7GxsSxfvpyIiIiyMY8++igBAQHce++9AFx//fVceOGFXHbZZUBJycrJyaG4uJjc3FwCAwM5ePAgvXr1YuvWrRhjysaUl5ycTKtWrfj222/p1avXb3I5nU4GDx5ctjVPeHg4CQkJNGrUiIMHDzJq1Cjmzp2Lv78/zzzzDAUFBfzzn/+s0h+jVnwXEZGTlVtYzCMzN+Ja+zFPe7+FCWqG51WfQHDN+H2iFd+rwEsvvcSMGTMA2LNnD1u3biU9PZ3+/fsTEREBQIMGDU7qNa21PPTQQyxfvhyHw8HevXvZv38/ISEhxz2nZcuWZQUL4NNPP2Xy5MkUFxeTlpZGUlISMTExvzln9erVJCUl0bdvyScyCgsL6d2790llFRERqWob9h7h7ikJjD78P27x/gpXeH8cV7wHdU7u92l1VfNK1gmuOJ0OS5cuZdGiRaxatYo6deowcOBA8vPzsdZWahNKT0/PsvlT1loKCwsB+Oijj0hPTycxMREvLy/Cw8PJz8//w9fy9/cv+3rnzp0899xzrFmzhvr163P99ddXeL61liFDhjBlypSTedsiIiKnhbWWd1Ym8+rcRF72foU+nmuhx604hj0JHl7ujldlNCerEo4cOUL9+vWpU6cOP//8M6tXrwagd+/eLFu2jJ07dwIlt+4A6tatS3Z2dtn54eHhJCYmAjBz5kyKiorKXjc4OBgvLy+WLFnCrl27TipXVlYW/v7+BAUFsX//fubOnVt2rHyGXr16sXLlSrZt2wZAbm4uW7ZsOZUfhYiIyJ+SkVPATe8l8OFXi/iqziP0NhtgxItw/rO1qmBBTbyS5Qbx8fG88cYbxMTE0L59+7LbdY0bN2by5MmMGjUKl8tFcHAwCxcuZMSIEVx22WXMnDmTl19+mVtuuYWRI0fSo0cPBg8eXHY16qqrrmLEiBHExcXRpUsXOnTocFK5OnfuTNeuXYmKiqJVq1ZltwMBxo0bx/DhwwkNDWXJkiW8++67jBkzhoKCAgCeeOIJ2rVrV0U/IRERkRP7dttB7v7kRzrlrWGe/yt4eflgrp4FLfu4O9ppUfMmvku1pT8nERGpSJHTxQuLtvDa0m3cH7iIWwvfwwRHwpgpUK+Fu+P9KZr4LiIiIm6xJzOXO6euJWn3AT4PmUq3w/Og40Vw8evg8+cW3q7uVLJERETktJi9LpUHp62nEYdYFfoaDQ79BAMfhP73gaP2TwuvMSWrsp/kE/eobredRUTEfXILi3nsyySmrtnD5aEHeKrwGTxzjsAVH0DkRe6Od8bUiJLl6+tLRkYGDRs2VNGqhqy1ZGRk4Ovr6+4oIiLiZkmpWdwx5Qd2HDzKK9HbuCD5Xxj/YLhmQcni32eRGlGywsLCSElJIT093d1R5Dh8fX0JC6t5m3eKiEjVsNby/qpdPDlnEw18HayIXU7YxjegZV+44n3wb+TuiGdcjShZXl5eZauqi4iISPWSebSQ+z7/iUWbDnB+O39e8HwV740LoNv1MPzf4Ont7ohuUSNKloiIiFRPq7ZncPcnazl0tIh/Dwrgsi1/w2Rsg/Ofg+431+gNnv8slSwRERE5acVOFy9+vZVXlmwjoqE/U88rIGLxtSWl6poZ0GqAuyO6nUqWiIiInJSUQ7ncNfVHEncd4vLYZjzRdCU+cx+GRu1KFhhtoCk+oJIlIiIiJ2HO+jTun7YOa+HlyyMZkfI8LPoA2p8PoyaDT113R6w2VLJERETkhPIKnTw2O4kp3++mc/N6vHpRc8IW3AJ7VkO/e+Hcv58VC4yeDJUsERER+UM/78ti4sdr2XYgh/EDWvPXmAK8PhkOuRlw2dsQfam7I1ZLKlkiIiJSIWstH67exeNfbSLIz4sPbupBv8Jv4N0J4FcfbpwLTbu6O2a1VanresaYeGPMZmPMNmPMA38w7jJjjDXGxJV77sHS8zYbY4ZVRWgRERE5vQ4dLWTcB4k8PHMjfVo3ZO6dfemX8l/47HpoEg23LFHBOoETXskyxngArwJDgBRgjTFmlrU26ZhxdYE7ge/KPRcJXAlEAU2BRcaYdtZaZ9W9BREREalK3+3I4O5PfuRgTgH/uKAjN3ZvjOOLm+Hn2dDlKrhwEnj6uDtmtVeZK1k9gG3W2h3W2kJgKjCygnGPA88C+eWeGwlMtdYWWGt3AttKX09ERESqmWKni/8s3MKY/67Gx9PB9Nv6cnO0A8fbw2DzHBj2FIx8VQWrkipTspoBe8o9Til9rowxpivQ3Fo7+2TPLT1/nDEmwRiToP0JRUREzry9h/MY89/VvPT1Vi7pGsbsO/vRqWgdTD4XslLgqs+h94SzegX3k1WZie8V/TRt2UFjHMAk4PqTPbfsCWsnA5MB4uLifndcRERETp95G9K4f9p6ip0uXhjdhYu7NoM1/4O590GDVjBmKjRs7e6YNU5lSlYK0Lzc4zAgtdzjukA0sNSUtNsQYJYx5qJKnCsiIiJukl/k5PHZSXz03W5iwoJ4eUxXWtbzhtn3QML/oO1QuPQt8A1yd9QaqTIlaw3Q1hgTAeylZCL72F8OWmuPAI1+eWyMWQrca61NMMbkAR8bY/5DycT3tsD3VRdfRERETsXmfdncMeUHtuzP4db+rfjr0PZ4FxyCD0ZD8groexcMfgQcHu6OWmOdsGRZa4uNMROB+YAH8La1dqMx5jEgwVo76w/O3WiM+RRIAoqB2/XJQhEREffJyi/i1cXbeGdlMoF+nrx/Yw/6t2sM+zfClCshez9cMhk6j3Z31BrPWFu9pkDFxcXZhIQEd8cQERGpVYqdLqau2cOkhVvIzC3k0tgw7o/vQOO6PrBpNkwfV7Lv4JUfQ1g3d8etMYwxidbauIqOacV3ERGRWm75lnSe+CqJLftz6BHRgPcujCS6WRBYC8v+DUuegKaxJQUrMNTdcWsNlSwREZFaatuBbJ78ahNLNqfTokEd3rg6lmFRIRhjoPAofDEBkr6AmNEw4kXw8nN35FpFJUtERKSWOXS0kBcWbeHD73ZTx8uDh87vwHV9wvHx9ACXq2Rh0cVPwIFNMOQx6HOn1r86DVSyREREaonCYhfvr0rmpa+3klNQzNieLfjLee1oGOADzmJY9yms+A+kb4J6LWDsp9BuqLtj11oqWSIiIjWctZYFSft5as4mkjNy6d+uMf+4oCPtmtSFonxIeBu+eQEO74LGHWHUfyFqFHioBpxO+umKiIjUYBtTj/DE7E2s2pFBm+AA3r2hOwPbB0NBNqx8CVa9Ajn7oVk3iH8K2g0HR2V21ZM/SyVLRESkBjqQlc9zCzbzWWIK9fy8eHxkFGN6tMCz4DAseQq+ewPyD0PEgJIrVxH9Ne/qDFPJEhERqUHyi5y8tWIHry3dTpHTxc3nRDBxUFuCig7Cooch4R0oOgrtL4B+90BYhUs4yRmgkiUiIlIDWGuZ9VMqz87bzN7DeQyLasKDwzsS7tgPi+6FHz8GlxM6XQZ974Ymke6OfNZTyRIREanmfth9iMdnJ7F292Gimgby3OWd6R2wD5beCRung8MLul5dshRDgwh3x5VSKlkiIiLV1N7DeTwz92dm/ZRK47o+PHtZDJcGp+Hxze2wZS54B0DvidD7dqgb4u64cgyVLBERkWrmaEExry/dzn9X7ADgjnNbM6HlHvxW3wbJK8CvPgx8CHrcAnUauDmtHI9KloiISDXhdFmmJabw7wWbSc8u4OLOITzcNpmGP4yHVWuhbigM+xfEXgc+Ae6OKyegkiUiIlINrNqeweOzk0hKy6J78wA+672L8KT/g82boX5Eyd6CnceAp4+7o0olqWSJiIi4UfLBo/xrziYWJO0nIsjBrJ6b6JT8Lmb5HgiOgkv/B5EXa3X2Gkh/YiIiIm5wJK+Il7/eynurkqnvkc/HHX+g94FPMD8dgLAecP5z0G6YFhCtwVSyREREzqBip4uPv9/NpIVbMHkZvBG2inOzvsCx8wi0HgT9/got+6pc1QIqWSIiImfIks0HePKrTeQc2MWTDRcT75iHIz0POo6Ac+6BZrHujihVSCVLRETkNNu8L5sn52xi99Z1/M1/LsP9lmFyXZiY0XDO3dC4vbsjymmgkiUiInKaZOQU8J+FW1i7ZgV3eH/JMJ/VGLwxcddDnzugfkt3R5TTSCVLRESkihUUO3l3ZTLfLJ7NjXY6T3r/iPUOwPS4C3pNgIBgd0eUM0AlS0REpIpYa5m3Po1FX03lirxPuNXxM06/BtDnH5jut4BfPXdHlDNIJUtERKQKrN+dyYLpbzEs8yOedyRTEBAC/Z/BI/Ya8PZ3dzxxA5UsERGRP2H/oWyWfPYq3VPe5a+ONLICWuIc/DI+na8ET293xxM3UskSERE5BXlHs/lu2gu02/4uV5qD7AtoR+7QJwiMuQQcHu6OJ9WASpaIiMhJcOUeZtOXkwjd9DYDyWKbXycODHmBkNgLtYCo/IZKloiISGXkpJM2/z8EbniXKJtLglcc+wbdS2Tv4e5OJtWUSpaIiMgfObyH7CWT8Fn3AU1cRSz26AX9/sqggefhcOjKlRyfSpaIiEhF0rdQtPw/ODZ8iq8LZtp+5HS7nSuGD6KOt359yonp3xIREZFf5B2GTV/iWvcpJnkFTrz4sPg89nS4iVtG9Cc0yM/dCaUGUckSEZGzW1E+bJ2P88dPMdvm43AVsceGML14FOubXsZdF/XhhuZaRFROnkqWiIicfVxO2Lmcoh8/gU1f4lWcQ6YN4kvnYL72GkDTyD5c0Lkpd7drjNEnBuUUqWSJiMjZwVpI/YH8Hz7BbpiGX8FBCqwfc53dWeo9kAbRgxjWqTnXtGqAl4fD3WmlFlDJEhGR2u3gNo4mTsH502cE5u7CWE+WurqywvcG/KPPZ0jncF5qUR8PfVJQqphKloiI1D7Z+ziyZiqFP35C46wk/KxhlSuSlX6349XpYgZ1accTYUG6FSinlUqWiIjUDvlHOPj95+T/MJXQwwkE4WKdK4LpdW7CRI+if1wMf2tSV8VKzhiVLBERqbFsUR6pa2aRlziFFhnf0Igikl1N+NR/NK7oS+ndoze3Ng5wd0w5S1WqZBlj4oEXAQ/gLWvt08ccHw/cDjiBHGCctTbJGBMObAI2lw5dba0dXzXRRUTkbGSdxexMmE9OwhRapS+mGUdJt0Es8D8fZ+RldOszmDEN/N0dU+TEJcsY4wG8CgwBUoA1xphZ1tqkcsM+tta+UTr+IuA/QHzpse3W2i5VG1tERM4mLqeLn9cuJ2vNFNrsn08rDpFt/Uj0P4eijpcS028EF9bTFSupXipzJasHsM1auwPAGDMVGAmUlSxrbVa58f6ArcqQIiJy9il2ulj3UyKHv59C631ziSSVQuvJ+jo92dJxFB36X86AekHujilyXJUpWc2APeUepwA9jx1kjLkduAfwBgaVOxRhjFkLZAH/sNauqODcccA4gBYtWlQ6vIiI1C6FxS7WbNhE5uopROybSyzbcFnDFr/OrO0wntYDx9KtXmN3xxSplMqUrIo+hvG7K1XW2leBV40xY4F/ANcBaUALa22GMaYb8IUxJuqYK19YaycDkwHi4uJ0FUxE5CySV+hk5cYdHPjuMyLS5tCLDXgYyx6ftmxqdx/hA66mQ6OW7o4pctIqU7JSgOblHocBqX8wfirwOoC1tgAoKP060RizHWgHJJxSWhERqRVyCopZunEPe9fMJDx1DgP5AR9TxEHvpuxuO4Gm/a6meWiku2OK/CmVKVlrgLbGmAhgL3AlMLb8AGNMW2vt1tKHFwBbS59vDGRaa53GmFZAW2BHVYUXEZGa43BuIQs3ppKcsIDwtDkMM98RaHLJ8apPRusxBPe9hkbNu9NI61hJLXHCkmWtLTbGTATmU7KEw9vW2o3GmMeABGvtLGCiMeY8oAg4RMmtQoD+wGPGmGJKlncYb63NPB1vREREqp/07AIWbEzj57UrCE+dwwWOVVxuDlHgWYfsiHicva4moNUAAjy0bKPUPsba6jUFKi4uziYk6G6iiEhNlXo4j3kb9rH2xx+I2DeHkY6VtHak4TSe5DQfRGCPMZj2w8HLz91RRf40Y0yitTauomP6vw4iIvKnJR88yryN+1i1LolW++Yz0uNbbnRsx3oacpv2wsY+gEfUSIL86rs7qsgZo5IlIiInzVrL1gM5zF2/j2XrtxORvoSRHiu5xWMjHl4uChpFQdfHMdGj8A8Kc3dcEbdQyRIRkUpzuSzzN+7jzSWbCN63nIs8vmW8x1p8vAspDmyJR5e/QvRl+AR3cHdUEbdTyRIRkRMqdrr4an0a73z9I30yZ/K293waeB/G5dcQR6frodMVeIbFgT4ZKFJGJUtERI6ryOlixg97+XTxdwzJns5Hnovx98rD1Wow9LoNR6tzQZ8MFKmQ/ssQEZHfyS9y8lliCnMWL+Pi3GlM8fwGT08XRI+CvnfhCI1xd0SRak8lS0REyuQWFvPxd7tZtXQuowunMcUjEaePL47YGzB9JkL9cHdHFKkxVLJERITs/CI+WJXM5hXTGFs8g5sdP1PkF4Tt9Tc8eo4H/0bujihS46hkiYicxQ7nFvLeim3sX/UR17pmMsGxh4K6TaHf03h1vQZ8AtwdUaTGUskSETkLHcwp4P1lSRR+/y5XM5swc5C8hu1h4Jv4RF8KHl7ujihS46lkiYicRfZn5fP+1z9QZ+1b3GDmU9/kkBvSAwa9hl/boVqCQaQKqWSJiJwFUg7l8smCFQRv+B8THUvwcxSSEzEMBt1LneY93B1PpFZSyRIRqcV2HjzKjLnzaLPlLe5yrAZPB/kdL4dz/0JA4/bujidSq6lkiYjUQlv2ZTH/q8+J2fUO9zjWUeBVh/yutxIw4E4CApu6O57IWUElS0SkFtmwJ5OVs9+jZ9r73OHYQY5PA3J6PkRA31vx8avn7ngiZxWVLBGRWmDtjjR++upN+qdP4VbHPg7VCeNov+cI6HENePm6O57IWUklS0SkhrLWkrA5me1zXmbQkWl0NYc5ENiRo4OfpH7nS8Dh4e6IImc1lSwRkRrGWsvqnzZyYOEkBuV8RXeTR0qDXuTF30dwu0FahkGkmlDJEhGpIay1fPvdanKX/If++YvxNC52hw7B+/wHCGsR6+54InIMlSwRkWrO6bKsWj4Ps/JFeheupsh4savlpYSPeICIxq3cHU9EjkMlS0SkmioudvLdwk/xT3iFc5wbyCKALe3H0ebCv9IusIm744nICahkiYhUM4UFBSTOeYvgdW/S1+4i3TRiY6f76XDBHXTwrevueCJSSSpZIiLVRP7RLH6a9TItNr9Db9LZ7dGCDd2eJnLIjTT28nF3PBE5SSpZIiJudvTQfn6e+Tytkz+mJ9ls8ooko/cTRB/lJycAACAASURBVA28HKNlGERqLJUsERE3yd63nR2znqFd6gy6UUiiby98Bt5DVM+hGC3DIFLjqWSJiJxhWTvXsverp2h7cCGR1vBd3fOoP+ReunXu4e5oIlKFVLJERM4Eazm0aTGZ85+l9ZHVOKwvS+pdSvPz7+Wc9h3cnU5ETgOVLBGR08nl4tAP0zm6+HnCcpNw2kBmN76JDhf9hSEtmrs7nYicRipZIiKnQ3EBGd++j+ubF2lcuIdsG8z0pvcQO/J2Lgxp5O50InIGqGSJiFSFgmxIXUvx7jUc3roK37Tvaeg8wgYbwdfh/0ffETcxqpHWuBI5m6hkiYicLJcTDmyCvQmQkoArJQGT/jMGiyeQ7WrCtyaGI+0uY+iFo7kyyM/diUXEDVSyREROJCsVUhJKS1UipK6FoqMA5DgCWetsRaLzErZ4dqBR+97079yeoW0b4eulNa5EzmYqWSIi5RXkQNqPvy1V2akAWIcXhwI7sNZvKHPym5FYHEFOnRYM7RLK8OgQbm/VEC8Ph5vfgIhUFypZInL2cjkh/effFqr0TWBdJcfrR5Af1osNtGN2ZlM+3VOP3FxPQoN8GdY9hGeiQ4gLb4CHQwuHisjvqWSJyNkjK61sHhV7S2/7FeaUHPOtB826QccLyawfw/xDTfliSwFrfszEZaFlwzpc0y+E4dGhdA4L0orsInJCKlkiUjsVHoXUH39bqrL2lhxzeEJIJ+g8BsLioFkcuwllXtI+5m7Yx9rdh4E02jUJYOKgtsRHhdAxtK6KlYiclEqVLGNMPPAi4AG8Za19+pjj44HbASeQA4yz1iaVHnsQuKn02J3W2vlVF19EBHC54ODm3972O5AE1llyvF5LaNELmsWVlKqQGPDyZduBbOau38fcpWkkpW0GILpZIH8b1p746BBaNw5w45sSkZrOWGv/eIAxHsAWYAiQAqwBxvxSokrHBFprs0q/vgiYYK2NN8ZEAlOAHkBTYBHQztpf/ub7vbi4OJuQkPDn3pWI1G7Z+8tdoUqAvWuhMLvkmE8QNIstu0JFs24Q0BgAay0bU7OYv7HkitW2AyW3Cru1rM/w6BCGRYXQvEEdd70rEamBjDGJ1tq4io5V5kpWD2CbtXZH6YtNBUYCZSXrl4JVyh/4pbmNBKZaawuAncaYbaWvt+qk34WInJ0KcyHtp9/e9juyp+SYwxOaREHMFb+WqoZtwPHrJ/xcLsuPuw8xb8M+5m3Yx+7MXBwGekY05NreLRkWFUKTQF83vTkRqc0qU7KaAXvKPU4Beh47yBhzO3AP4A0MKnfu6mPObVbBueOAcQAtWrSoTG4RqY1cLsjYWu62XwLs3/jrbb+gFiVlquf4kn+Gdgav3y/06XRZ1iRnlhWrfVn5eHkY+rZpxISBrRkS2YSGAT5n+M2JyNmmMiWropmev7vHaK19FXjVGDMW+Adw3UmcOxmYDCW3CyuRSURqg5z039/2KzhScswnsOS23zl3/zqXKiD4uC9V5HTx7fYM5m1IY8HG/WQcLcTH08GAdo25v1N7BnVoQpCf1xl6YyIilStZKUD5reLDgNQ/GD8VeP0UzxWR2q4oH1a/ConvwuHdJc8Zj5Lbfp0u/bVQNWz7m9t+FckvcrJi60HmbkhjUdJ+svKL8ff24NwOwQyPDmVg+8b4++hD1CLiHpX522cN0NYYEwHsBa4ExpYfYIxpa63dWvrwAuCXr2cBHxtj/kPJxPe2wPdVEVxEahhrYcs8mPcgHNoJrQdBj3ElpSq0M3hXbsL50YJilmw+wNwN+1j68wGOFjoJ8vNiSGQIw6NDOEfb2YhINXHCkmWtLTbGTATmU7KEw9vW2o3GmMeABGvtLGCiMeY8oAg4RMmtQkrHfUrJJPli4PY/+mShiNRSB7fBvAdg20Jo1A6umVFSsirpSG4RX/+8n7kb9rF8SzoFxS4aBXgzsmsz4qNC6N1a29mISPVzwiUczjQt4SBSixRkw/J/w6rXwNMXBj4APW8FjxPPjcrIKWBBUkmx+nbbQYpdtmQ7m6iSK1bazkZEqoM/u4SDiMjJsRbWfwYLHoacfdDlKhj8CNRt8oen7TuSX7qGVRrf7/x1O5ub+kUQHxVC57B6OFSsRKSGUMkSkaqV9hPMuQ/2rIamXWH0h9C8+3GH78nMZe6GtHLb2UDb4AAmntuG+OhQbWcjIjWWSpaIVI3cTFj8OCS8A3UawkUvQ5erK/yE4L4j+XyeuIe5G/axMbVkLePoZoHcO7Qd8dGhtAnWdjYiUvOpZInIn+MshsR3YPETJXOweo4vmXvlV+93Q/dk5vL6su18npBCodNFbIt6/P38jsRHazsbEal9VLJE5NQlr4S598P+9RDeD4Y/C00ifzdsR3oOry3dzoy1e/Ewhiu6h3Fr/9YqViJSq6lkicjJO7IXFv4TNnwOQc3h8vcgciQcM3dq875sXlmyja/WpeLt6eC63uGM69+KkCDtFSgitZ9KlohUXnEBrHoFlj8HLicMuB/63v27hUTXpxzhlSVbmb9xP/7eHozr35qb+0XQSPsFishZRCVLRCpn87ySBUUP7YQOF8KwJ6F++G+GJO7K5OXF21i6OZ1AX0/uGtyWG/qGU6+Ot3syi4i4kUqWiPyxg9tg/oOwdUHJau1XT4c2g8sOW2tZtSODVxZv49vtGTTw9+Zvw9pzbe+W1PXVhswicvZSyRKRihVkl9wWXPVqyWrtQ5+AHreCZ8lVKWstS7ek88ribSTuOkRwXR/+cUFHxvZsQR1v/dUiIqK/CUXkt35ZrX3hPyE7DTqPhfMeLVut3eWyLNy0n1cWb2P93iM0q+fH4yOjuDyuuTZmFhEpRyVLRH5VfrX20C5wxfvQvAcATpflq/VpvLp4G5v3Z9OyYR2evTSGi7s2w9tTmzOLiBxLJUtEfl2tPfFd8KsPI16CrteAw0GR08XMH1N5bck2dhw8SpvgAF4Y3YULY0Lx9FC5EhE5HpUskbOZywkJb/+6WnuPcaWrtdenoNjJ52t28frS7aQcyiMyNJDXr4plWFSINmkWEakElSyRs9Wub0tuDZat1v4MNIkir9DJ1JU7eXPZDvZl5dOleT3+76IoBnUI1kbNIiInQSVL5GyTlQoLHi5ZrT0wDC5/FyIvJqfQyYfLtvPWih0czCmkR0QDnru8M33bNFS5EhE5BSpZImeL4oKS5RiWPweuYuh/H5zzF444vXhv8TbeXrmTw7lF9GvbiInntqFnq4buTiwiUqOpZImcDY5drX3oE2T6NON/S3bw/re7yC4o5ryOwUwc1JYuzeu5O62ISK2gkiVSm2VsLylXWxdAw7Zw9XQOBPflvyt28OHqxeQXOzk/OpQJ57YmqmmQu9OKiNQqKlkitVFBDiz/929Wa9/b/lre/GYPU99dgtNlGdm5KRPObU2b4LruTisiUiupZInUJtbC+s9h4cNlq7WndPsbr6zJYdpXKwG4NDaM2wa2pmVDfzeHFRGp3VSyRGqLtHUw9z7YvQpCu5Ay5A2e31SPma9vwtPDwZgeLbh1QGua1fNzd1IRkbOCSpZITZebWbKYaOI74Fef1P7P8q/UWL6acgBfz3xuOieCW/q1IjjQ191JRUTOKipZIjWVy1lSrBY/AflZpHe8lsdzRjJrQR4BPplMGNiam85pRQN/b3cnFRE5K6lkidRE5VZrzwrpxVOBNzDlh7rUq1PMPUPacV3vcILqeLk7pYjIWU0lS6QmyUqFhf+E9Z9RUCeE1+o9xIvJUTQK8OGB4a24uldLAnz0n7WISHWgv41FaoLS1drt8udwOYuYVudKHskcSlBgPR4Z0Yoru7fAz9vD3SlFRKQclSyR6m7LfOy8BzCZO1jt1Yv78kZjfcL5xyWtuaxbGD6eKlciItWRSpZIdZWxHde8B3BsXcAeRzP+UXg/KQF9uCu+DSO7NMXLw+HuhCIi8gdUskSqk+JC2L0KZ9Is+OE9ClweTCoay7cNL+PWkR25oFMoHg7j7pQiIlIJKlki7nb0IGxdiGvzPFzbFuFZlIMTT2Y5+zCz0c1cfV5PHuzYBIfKlYhIjaKSJXKmWQv7N8CWebg2z8PsTcRgyaAei4q7s9LRDe92gxjRvS3vt2uMMSpXIiI1kUqWyJlQlAc7l5cVK0d2KgBJthULi0ex2qs7YR17Et+pGc+1bYSvlyazi4jUdCpZIqfLkRTYMr/k04E7l2OK88g3vix3dmKh80LW+XQnNroD8dGh3N6qId6emsguIlKbqGSJVBWXE/b+AFvmlZSr/esBOOARwrzC/ix0diXZvyuDuzbn0ugQng5voEnsIiK1mEqWyJ+RnwXbF5eUqq0LIPcgLuPBZq9IvigewyJnLEX12jC8Wyh/iQ6hS1g9TWAXETlLqGSJnKyM7aVXq+aV7CHoKibfM5DvPWL5vDCaZa4YGtcNYXhcCC9FhxAZGqjJ6yIiZ6FKlSxjTDzwIuABvGWtffqY4/cANwPFQDpwo7V2V+kxJ7C+dOhua+1FVZRd5MxwFsHuVaXzq+ZBxjYAMuq0YrH3xXx6JJIf8tvSoWl94nuEcGenENoE13VzaBERcbcTlixjjAfwKjAESAHWGGNmWWuTyg1bC8RZa3ONMbcBzwKjS4/lWWu7VHFukdOrdO0qtswruR1YkIXL4c3uwFhm+w5i6pFIUvKD6dqiHsP7hPB8VCgtGtZxd2oREalGKnMlqwewzVq7A8AYMxUYCZSVLGvtknLjVwNXV2VIkdPOWti/8ddJ6ylrAEuRX2M2Bg7ks6xIZhxpR36eL93DG3BzvxCGRYcQGuTn7uQiIlJNVaZkNQP2lHucAvT8g/E3AXPLPfY1xiRQcivxaWvtF8eeYIwZB4wDaNGiRSUiiVSBsrWrSpZZICsFgJyGnVgTeiPvHezAskOheBzxoE+bRjw8OIQhkU1oFODj5uAiIlITVKZkVTRj11Y40JirgThgQLmnW1hrU40xrYDFxpj11trtv3kxaycDkwHi4uIqfG2RKnFkL2wtLVU7lkFxHtbLn4wmfVgWeDWT01qzea8/3p4OBrRrzHNRIZzXsQlBdbzcnVxERGqYypSsFKB5ucdhQOqxg4wx5wF/BwZYawt+ed5am1r6zx3GmKVAV2D7seeLnBYuF6T+8OunAfeVfAbDFdSClPDLmFvYmf/ubsrBbVDH24NzOwRzR3QI57YPxt9HH74VEZFTV5nfImuAtsaYCGAvcCUwtvwAY0xX4E0g3lp7oNzz9YFca22BMaYR0JeSSfEip08Fa1dhHDjDerIl+l6mZUcxZacfR/e7CPT15LzIJsRHhdC/XWNtZyMiIlXmhCXLWltsjJkIzKdkCYe3rbUbjTGPAQnW2lnAv4EA4LPS9YB+WaqhI/CmMcYFOCiZk5VU4TcS+TMytv+6xMKub8FVBL71KGw1mJ/8evLRwbbM3V5AQbGLhv7eXNSlCfHRofTWdjYiInKaGGur1xSouLg4m5CQ4O4YUt39Zu2q+ZCxteT5xh3ICz+Pbz3i+Cg1hBXbD1HktDQJ9CE+KoT46FB6RGg7GxERqRrGmERrbVxFxzTpRGqen7+CmbdD3iHw8IbwfmTF3MCi4s58tt2T777JwGWheYM8bugbQby2sxERETdQyZKaw1r45j/w9ePQtAsHBz3Pl9ntmb05i8S5h4AjtG7sz4SBbYiPDiGqqbazERER91HJkpqhKB9m3QHrP+Vw65Hccvg61kzLB3YRGRrIX4e0Y7i2sxERkWpEJUuqv+x9MHUs7E1kU+TdXLK+J/X84MHhHYiPDqFlQ393JxQREfkdlSyp3lLXwpSx2PwjzGj7DPf80Jy4lvV4/epuNK6rlddFRKT6UsmS6mvDdPhiAq46DXmk0fN8sD6QsT1b8OiIKC27ICIi1Z5KllQ/LhcsexqWPUNeSHfGZk1k/S5vnrwkiqt6tnR3OhERkUpRyZLqpfAozBgPm2aRGn4pF+wchYeXD1PGdaN7eAN3pxMREak0lSypPg7vgaljsPs38k3E3Vz7c3eim9bjzWu60bSen7vTiYiInBSVLKke9nwPU6/CFufxasgTPLcpnEu6NuOpUZ20n6CIiNRIKlnifj9OgS/vpCigKbc7HmFRcn3+cUFHbjonQouJiohIjaWSJe7jcsLX/wcrX+Rwk16MPDCOw9TlvRu70q9tY3enExER+VNUssQ98rNg+i2wZR5JzS7n4h0jCA+ux6xr47S4qIiI1AoqWXLmZe6EKWOwB7cwrcnd3Lu9B8OimvD8FV0I8NG/kiIiUjvoN5qcWTtXwKfX4nK5+Gfdx/hwVwR/Oa8ddwxqg8Oh+VciIlJ7qGTJmZPwDsy5l7y6LRmbczdb8oKZfE0XhkaFuDuZiIhIlVPJktPPWQzzH4Lv3yS1UV8uSLuRoHoN+WJcHG2b1HV3OhERkdNCJUtOr7xD8NkNsGMJ3zQezbV7RnBOuya8fGVXgup4uTudiIjIaaOSJafPwa3w8Wjs4d28EfgXntnTnVsHtOK+YR3w0PwrERGp5VSy5PTY9jV8dgPFxoPbPR5laWZrXrwyhpFdmrk7mYiIyBmhkiVVy1r47k2Y/yBH6rbhksyJFAQ0Z9pt3YhuFuTudCIiImeMSpZUneJCmHMv/PAem+v155J91xMd0ZTXroqlUYCPu9OJiIicUSpZUjWOZsCn18CulcysO4a7913ANb0jePjCSLw8HO5OJyIicsapZMmftz8JpozGlb2fJ33+wvuZPXhqVDRX9mjh7mQiIiJuo5Ilf87muTDtZgocflxX9E+2OzowdVws3Vo2cHcyERERt1LJklNjLax8EbvoUQ74d2BkxgSahLXiy2viCAnydXc6ERERt1PJkpNXlA9f3gXrppLgP5BrMq7jgtjWPHlJNL5eHu5OJyIiUi2oZMnJyd4Pn1wFKWt412csjx+6kL9fGMkNfcMxRguMioiI/EIlSyov9UeYOhbn0Uz+xj0sLurN+zfG0rdNI3cnExERqXZUsqRyNn6BnTGeox5BjM57GGdwJ768No7mDeq4O5mIiEi1pJIlf8xaWPYsLP0XyX5RXH5oIj07deDfl8dQx1v/+oiIiByPfkvK8RXmwswJsHEGX/sMYsLh67hzWDQTBrbW/CsREZETUMmSih3ZC1PHYNPW8aK5hrfyL+S1a7syuGMTdycTERGpEVSy5PdSErBTx1KUl8PtRfeyvf45fHFtHG2CA9ydTEREpMZQyZLf+ukT7Kw7yPRoyJW5jxDWriszruxKkJ+Xu5OJiIjUKCpZUsLlgsWPwTeT2OjViWuybmfMwK78dWh7PByafyUiInKyHJUZZIyJN8ZsNsZsM8Y8UMHxe4wxScaYdcaYr40xLcsdu84Ys7X0f9dVZXipIgXZJQuMfjOJGY4hjM1/gMfHDuC++A4qWCIiIqfohFeyjDEewKvAECAFWGOMmWWtTSo3bC0QZ63NNcbcBjwLjDbGNAAeAeIACySWnnuoqt+InKJDyTBlDK70zTzhvJ75Phcx9ebuRDYNdHcyERGRGq0yV7J6ANustTustYXAVGBk+QHW2iXW2tzSh6uBsNKvhwELrbWZpcVqIRBfNdHlT0teif3vIPIy9nBtwX1saj6GL+/sp4IlIiJSBSpTspoBe8o9Til97nhuAuaezLnGmHHGmARjTEJ6enolIsmf9sP72PdHklbox/m5j9Km1wjev6kHDfy93Z1MRESkVqjMxPeKJuXYCgcaczUltwYHnMy51trJwGSAuLi4Cl9bqoizGBY+DKtfY42jC7fl3cH9l/bkirjm7k4mIiJSq1SmZKUA5X8DhwGpxw4yxpwH/B0YYK0tKHfuwGPOXXoqQaUK5B2Gz2+E7V/zvh3O644b+O+tPYhtUd/dyURERGqdypSsNUBbY0wEsBe4EhhbfoAxpivwJhBvrT1Q7tB84F/GmF9+iw8FHvzTqeXkZWzHfjwaV+YO/l50M5ubjeKLq7vRJNDX3clERERqpROWLGttsTFmIiWFyQN421q70RjzGJBgrZ0F/BsIAD4r3dNut7X2ImttpjHmcUqKGsBj1trM0/JO5Pi2L8F+eh05RXBT/kOEdxvC1Iuj8fH0cHcyERGRWstYW72mQMXFxdmEhAR3x6gdrIU1b2Hn3s9OE8YN+fdw44UDubZ3S23wLCIiUgWMMYnW2riKjmnF99rKWQRz74OEt1lKNx42d/Pvm/rSu3VDdycTERE5K6hk1UZZqdjpt2CSv+G14ouY0+gmpl7Xg7D6ddydTERE5KyhklVbZO+DpFmQ9AV217cUG0/uK7yNougr+Oyyzvh5a/6ViIjImaSSVZMdU6wMlr1e4Ux3Xcq0oj6MHjaQ8QNaaf6ViIiIG6hk1TQVFKsUr3CmOy9lVnEPsrxaMyw2hEmxzeiq9a9ERETcRiWrJqigWO3xbMm04lHMdvYkz7ctw3uG8HR0CLEt6uNw6MqViIiIu6lkVVe/FKuNM7C7V2Gw7PYoKVZfOXviCmhPfFwIk6JDiW4WqFuCIiIi1YxKVnVSQbHa5dGS6UWj+MrVE88mHRneO5RXo0No1yRAxUpERKQaU8lyt9JiZTdOh92rMViSHS2YUVqs/JtFER8dyn+jQ4ho5O/utCIiIlJJKlnucJxi9UXRKObYntRvEUN8dAjvRYfQrJ6fu9OKiIjIKVDJOlMqKFY7TXNmFo1inu1F41adiY8O4aOoJgTX1abNIiIiNZ1K1umUlQabZmE3zigrVjtozqyiUSygN03bdiE+OoSpHZtQ39/b3WlFRESkCqlkVbXSYuXaMAOzp6RYbac5XxaN4mtHH5q370p8dAifdgimrq+Xu9OKiIjIaaKSVRVKi5VzwwwcvxQr25zZxaNY6tmXiI6xxEeHMr5dY21vIyIicpZQyTpVpcWqeP10PFK+K7kVaMOYXTyKb7z70jo6juHRoUxo0xAfTxUrERGRs41K1skoLVZF66fjWa5YfVU8ilV+/Wgf3Z346BDuiGiAp4fD3WlFRETEjVSyTqS0WBWum47X3pJitdMVxlfOUST4D6BDTHeGR4dwl7azERERkXJUsipSWqwK1k3Hu1yxmuMcxU9BA+kY04Ph0SHc3SxIq66LiIhIhVSyflFarPJ/moZP6vclC4SWFqtNDQYR2bkH50eHcre2sxEREZFKOLtLVlYaNmkmeT9Nxy+tpFjtKi1W2xufR1SXnvx/e2cftFk5x/HP99k2u6skJavakEopaxVbmlWk1QtqNU2MJAZhUomKvEsT1RCKRmPEoLyW0sugQaLQGkVITbW1St5WJXrbfX7+uK5HxzP78rzc5z7nXOf7mbnnuc/LfT+/z7muc87vvs51rrNkp7kc68fZGGOMMWaS9C/JWvkQsfRc/nPdt5lz97WIYPnollw2ehB3zF3MTgt24xA/zsYYY4wx06R3SdaDoyP863unsmLVHC4fPYi7ttiX+c/ZlUP9OBtjjDHGDJDeJVmz1p/JOTufz9wnb8HhfpyNMcYYY2qid0kWwNEv363pEIwxxhhTOB4x0xhjjDGmBpxkGWOMMcbUgJMsY4wxxpgacJJljDHGGFMDTrKMMcYYY2rASZYxxhhjTA04yTLGGGOMqQEnWcYYY4wxNeAkyxhjjDGmBpxkGWOMMcbUgCKi6Rj+D0l/A24fwr/aFPj7EP5PU5TuB+U72q/7lO5ov+5TuuMw/J4SEU9c3YLWJVnDQtLSiHhu03HURel+UL6j/bpP6Y726z6lOzbt58uFxhhjjDE14CTLGGOMMaYG+pxkndN0ADVTuh+U72i/7lO6o/26T+mOjfr1tk+WMcYYY0yd9LklyxhjjDGmNpxkGWOMMcbUgJMsY4wxxpgacJJljDHGGFMDTrImgaTHS1qv6TjqRlKx9ULSRoX7bVp6HZX0FElPyO/VdDyDRtJOkjZvOo66kDRf0mJJM5uOpS4k7SDpGEmbNh1LHUiaJ2nDpuOoi0Ge64s92QwSSbMkfRX4FnCmpLlNxzRIJG0g6WxJ72k6lrrIjp8DzieV4WZNxzRIst9ZwDeBMyQ9vemY6kDSXsB1wAkAUdDt0bkMTwMuAp7TdDyDRImZkk4FLgHmAes3HNbAyWV4OnAlqY7+o+GQBoqkOZLOAb4DfFrSy5qOaZBImi3pXOArwMmSVvuonMngJGtiHAysBF4KzACOk/SiZkMaDLlF4Dzg6cChkraPiFFJMxoObWBI2gb4IfAf4FXAFsAbJM0ooSVE0tOAa4FHgP2AbUiexbT0VDzuAb4PbC9pz3HLOoukLYCbgNnAThFxaWVZ5/1yMjwP2Bp4VkR8ISL+DWX4AUhaAPwaWAU8CbgD2Dsv67xjPie8C3gIWAisAHbJy0rwGyElxvcBS/Ls4yTtPp3vdZI1MV4M/D4iHgI+AiwHXiFpTrNhTZ+IWAF8knRyPo/kR0SsajKuAfN34EMRcWxE3AecCrw2IlaV0BISEbcBB2a/B4GrSQfCYlp6Kh47kMrz68AR45Z1loi4E/gdcH1EPCDpBZJ2kzS7BL/MIuCWiLhX0sskHSlpW6CUy9vLgMUR8e5cZj8FtoVi6ugq4Fmkc+Eq4C/AY0s4DwJExCipBfnqiFgJfA7YEzhY0uypfq+TrLVQ6btzAbCLpMdExHLg58AosH9jwQ2Ayq+PH+Wd5vPAPEkvzctLOfjdD1wBkPuB/Bm4ITcNd/4XGEBE3JRb5r4IvB3YVdLpkp7XcGgDoVJOtwN/BH4GPCzpWEmLm4ts+lRajd8BfFjSxcBHgfcCn5H0/MaCGwCVsvstsETSCcDbgGcAJwHHNBXbIImIeyJiWcV3g/yqlnHX+TLwRkk/B44CNgfOBg5pNKoBkMvtCmDvfO6/n5Q4zwL2mur3OsnKjO0Y1ZNuzmwBbgH+yqMV6UZSU2lnMvg1+MW4v38hZe8n5umVXUpCVucIySMiHs7vHyH9GhuNiAe69AtzTX5jIV1RZAAAB1lJREFU83KifEZEbAIcRro8uleXOhivpQzHymkhsCIi7gCeTDpJb726z7SRNeyHqySNRMRvSY8AWRoRi0hluBzYvSs/eNZ2nAGuB34AHBwR+0bE0aTjzSJ1qJ/r2vbDsVXy3yuAV0K3rgyszS8iLgTeCNwcEfMi4jXAj4AXSNpouJFOjXUcYy4hndcvAH5M6lt3P6m70JRwkgXkHfwaSZus4aR7G/ALYB9JW0fEvaTr7jsOM86pMgG/sfUEfA24S9KbcwvBomHFOR0m6phZTNqZkHSgpB1qD3CarMuvMu83efpB0g+BzXJi2XrW5lg5IN4IHCHpVlLfiW8Aj4P2X5KZYB09KSJOgtQyAtwLbJ4vX7SaCdTRUeBCYEdJ8/Ps+4C7gb8NL9KpM5EyrPw4v5HUYv6MoQU4TSZYR/8EbKLUjxBSHZ2Rz4utZgJ1dBnwOlJDwx4RcTbpBo3tp/o/nWQBEXE3aUc/cQ3LHyB1tl0GnCtpH9Klwt8NK8bpsC6/ynqRXZeSmoA/Quq82Xom4lhpDRgBni3pUuCtpM7UrWYyZQggaVdS583f1x/dYFibY+WAuJKUSB4UEYeQLl9sJmnW0AKdIuvwG81//3fgl7QQOIB0ma31TKSORsQPgVOAj0o6gNQ/8uHhRDh9JrofZh4Cngn8s9agBsgE/WYBN5Puvnsu6dLvii60JE+wjj4SEX+IiH9I2gPYmdRaN+V/2ssXKfOuTm9FarGav47PHQl8CTiiaYea/N5EOqi/ummHGh3/DPyB1Fm8cY9B+pHuavoUaZiDQ5t2GLQj+aH2lelZTTvUUIYbAGfkMmz1fjiNffDlpBtuDmvaoS7HvO4FwPzx9bZNrynW0W2AzwKXA4c37VBH+ZEaGa6fbh1V/rLeIulg4NqIuF3S+4CFEXHAatYbiUebgdc5vy1Mwk8REZKeFKlv1tj8GdHy/gQTdczrrg8sivSLemxeqx0n6bcesCAillbmtbqOwuQc8/r/59R2xyn4LYiI6yrTihYfrCd7nFnN/FaXH0x6P2y9z3imUEdnRqUrQtudp3AunBfpRrfp0XSWOcRs9iXA+uOmbyDdUXc28Po8/wZgSX6vyvoj46Zn1B1zw37rNe00BMeZTTu5DKfn2LZX6WVYg99I005DcCz9XFG637T2wcY3yBA2+Fhr3S+Bk/P72aS7kp4KbEy6dPSJvOww0jhD6419vlqJgO2adqrZb9umnYZRhrToRF26X12OTTv1qQxLL78+ONpv8ufCQeyDjW+YYWz0/H5XUl+jLfP010m32F4DvCXPG8l/rwfeNu67NgbOzOs/rmm3Pvj1wbF0vz442q/bfn1wtF9zfo1vnCEVwPGkZsJbga/leacAF49tRNLjct6U329JpYkQeDcp492naZc++vXBsXS/Pjjar9t+fXC03/D9Gt8oA97AI4y7xk+6Bfoa0sCFi0i30y4kDZf/edJdPEtIo7h/kTQQWTUr3g94Py3ov1O6Xx8cS/frg6P9uu3XB0f7tcev8co+wI1e3VibVd4fAXygMv1O4Kr8fhvgg8D5wL5NO/TZrw+Opfv1wdF+3fbrg6P92uXX+AYb8AafBXycdJ31XGAfUnZ7UWWdTUkDGr51Dd/XqrtdSvfrg2Ppfn1wtF+3/frgaL92+nV+xPfIWyvzFtI11t1Ig6R9CrgK2FDSCZK2Al4IXEa66+B/KD8MOlo2zkfpflC+Y+l+UL6j/brtB+U72q+dfp0bjHRs6P6xDS5pfx7d0HsByyLiq3nZeaRbNr8EvBnYG7gTOCoi7hx+9OumdD8o37F0Pyjf0X7d9oPyHe3XEb9hNJcN6kWleY/UaW0+6ZlCp5IeFHsrcExlneeRnvo+M08/tbJMtK85tGi/PjiW7tcHR/t1268Pjvbrjt/YA3M7QUSM5qa+k0h3DfwJOD4ilko6ENgdOEzSX4ErSYON/STy0P+RnrBdfYxKq5rxSveD8h1L94PyHe3XbT8o39F+3fHrVJ8sSXsC3wTuJw0WtgDYTul5dJcDlwCPAE8AvkvqHHfW+O+Jlj6nrnQ/KN+xdD8o39F+3faD8h3t1yG/JpsEJ/sCXgGMAlvl6ROBj5EfBQPsAdwFPBaYW/lcax5P0We/PjiW7tcHR/t1268Pjvbrjl/jAUxh418EnJbfzyPdvnk4MCfPW1jd4LTsWnPf/frgWLpfHxzt122/Pjjarxt+nbpcmPkAsJ+k7SJiOfAr0vXZjQAi4pdjK0aiVbehToDS/aB8x9L9oHxH+2U66gflO9ov02a/zg3hACDpZGDniNhf0gxgw4i4p+m4BkXpflC+Y+l+UL6j/bpP6Y72az9dbMkC+AywQtLGpCT2nrExNQqhdD8o37F0Pyjf0X7dp3RH+7WcTrZkGWOMMca0na62ZAGPDo9fKqX7QfmOpftB+Y726z6lO9qvvbglyxhjjDGmBjqbHRpjjDHGtBknWcYYY4wxNeAkyxhjjDGmBpxkGWOMMcbUgJMsY4wxxpga+C/u6I9IIQekFQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(expected_interaction_rates, label=\"expected rate\")\n",
    "plt.xticks(range(10), labels=new_model_names, rotation=30)\n",
    "plt.plot(estimates_matching\n",
    "         , label=\"actual rate\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Estimate Click Rate: Propensity Weighting\n",
    "\n",
    "First, let's use the propensity matching estimate:\n",
    "$$r' \\approx \\frac{1}{n} \\sum_{i=1}^{n} y_i \\frac{P'(w) }{P(w)}$$\n",
    "\n",
    "Here:\n",
    "- $y_i$: user interaction\n",
    "- $P'(a|x,b)$: selection probability of the offline model being evaluated\n",
    "- $P(a|x,b)$: logged probability of the production model (random serving)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use the same context ids as logged data:\n",
    "df_new_models_weighting = df_random_serving.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sample_prior_for_context_n_model(context_id, model_priors, selected_ad):\n",
    "    # get ad interaction priors for the given context\n",
    "    interaction_priors = user_context_priors.get(context_id)\n",
    "    \n",
    "    # get the selection prior for the given model based on interaction priors\n",
    "    selection_priors = model_priors[np.argsort(np.argsort(interaction_priors))]\n",
    "    \n",
    "    # get prior of the selected ad\n",
    "    selected_ad_prior = selection_priors[ads.tolist().index(selected_ad)]\n",
    "    return selected_ad_prior\n",
    "\n",
    "for model_name, model_prior in zip(new_model_names, new_model_priors):\n",
    "    df_new_models_weighting.loc[:, model_name] = df_new_models_weighting.apply(lambda x: sample_prior_for_context_n_model(x[\"context_id\"], model_prior, x[\"selected_ad\"]), axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>log_id</th>\n",
       "      <th>context_id</th>\n",
       "      <th>selected_ad</th>\n",
       "      <th>user_interaction</th>\n",
       "      <th>model_0</th>\n",
       "      <th>model_1</th>\n",
       "      <th>model_2</th>\n",
       "      <th>model_3</th>\n",
       "      <th>model_4</th>\n",
       "      <th>model_5</th>\n",
       "      <th>model_6</th>\n",
       "      <th>model_7</th>\n",
       "      <th>model_8</th>\n",
       "      <th>model_9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>57603</td>\n",
       "      <td>34a410fe-ca42-4c33-8306-476705e65eb1</td>\n",
       "      <td>context_8949</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>39408</td>\n",
       "      <td>303934cf-d3be-4d31-8173-cbfbf48ac205</td>\n",
       "      <td>context_1410</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35340</td>\n",
       "      <td>a2e8a13f-91f3-4d93-90e2-13284c86febc</td>\n",
       "      <td>context_5801</td>\n",
       "      <td>ad_1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>15035</td>\n",
       "      <td>ba6fb330-ef2f-4081-83df-f44a8271f16b</td>\n",
       "      <td>context_3822</td>\n",
       "      <td>ad_0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.1</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.2</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.45</td>\n",
       "      <td>0.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>76126</td>\n",
       "      <td>0250c20f-87be-405f-aed0-820a68b99f08</td>\n",
       "      <td>context_5041</td>\n",
       "      <td>ad_2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.8</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.6</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.4</td>\n",
       "      <td>0.3</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     log_id    context_id selected_ad  \\\n",
       "57603  34a410fe-ca42-4c33-8306-476705e65eb1  context_8949        ad_1   \n",
       "39408  303934cf-d3be-4d31-8173-cbfbf48ac205  context_1410        ad_1   \n",
       "35340  a2e8a13f-91f3-4d93-90e2-13284c86febc  context_5801        ad_1   \n",
       "15035  ba6fb330-ef2f-4081-83df-f44a8271f16b  context_3822        ad_0   \n",
       "76126  0250c20f-87be-405f-aed0-820a68b99f08  context_5041        ad_2   \n",
       "\n",
       "       user_interaction  model_0  model_1  model_2  model_3  model_4  model_5  \\\n",
       "57603                 0      0.8      0.7      0.6      0.5      0.5      0.4   \n",
       "39408                 0      0.8      0.7      0.6      0.5      0.5      0.4   \n",
       "35340                 0      0.1      0.1      0.2      0.2      0.3      0.3   \n",
       "15035                 0      0.1      0.1      0.2      0.2      0.3      0.3   \n",
       "76126                 0      0.8      0.7      0.6      0.5      0.5      0.4   \n",
       "\n",
       "       model_6  model_7  model_8  model_9  \n",
       "57603      0.4      0.3     0.20      0.2  \n",
       "39408      0.4      0.3     0.20      0.2  \n",
       "35340      0.4      0.4     0.45      0.6  \n",
       "15035      0.4      0.4     0.45      0.6  \n",
       "76126      0.4      0.3     0.20      0.2  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_new_models_weighting.sample(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that for each log entry, we've computed the probability of the same ad selection as production model from all new models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# match and estimate:\n",
    "estimates_weighting = []\n",
    "for i in range(len(new_model_names)):\n",
    "    model = \"model_{}\".format(i)\n",
    "    \n",
    "    # the logging policy was random so we know P(w) = 1/3\n",
    "    estimate = (df_new_models_weighting[\"user_interaction\"] * df_new_models_weighting[model] / 0.333).sum() / df_new_models_weighting.shape[0]\n",
    "    estimates_weighting.append(estimate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlkAAAFECAYAAAAUWCufAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeVxVdeL/8deHTRRwRUVFxX0BERF3Tcs998b2fuk0ZTXVNDU1bTNTOVa2WlaT+W3ady0Vt1zK3VwgTXNHRUVRURRkh3s/vz9gGHJMUcHD8n4+HvMYzj3nXN4fLnLfnXPu5xhrLSIiIiJSsjycDiAiIiJSEalkiYiIiJQClSwRERGRUqCSJSIiIlIKVLJERERESoFKloiIiEgp8CrORsaYIcAbgCfwnrV28m9sNxaYAXSx1sYYY0KAHcCugk3WWWvvOd/3CgwMtCEhIcUKLyIiIuKk2NjYE9bauudad8GSZYzxBN4GBgIJwEZjTLS1dvtZ2wUAfwLWn/UUe621EcUNGxISQkxMTHE3FxEREXGMMebAb60rzunCrkCctXaftTYH+BIYdY7t/gm8BGRdUkoRERGRCqQ4JasRcKjIckLBY4WMMZ2AxtbaeefYv5kxZpMxZoUxps+lRxUREREpP4pzTZY5x2OF9+IxxngAU4Dx59guEWhirT1pjOkMzDbGhFprU3/1DYyZAEwAaNKkSTGji4iIiJRdxSlZCUDjIsvBwJEiywFAGLDcGAMQBEQbY0Zaa2OAbABrbawxZi/QGvjVRVfW2unAdICoqKj/uZlibm4uCQkJZGXpTGRZ5evrS3BwMN7e3k5HERERKROKU7I2Aq2MMc2Aw8BNwC3/WWmtTQEC/7NsjFkOPFLw6cK6QLK11mWMaQ60AvZdbMiEhAQCAgIICQmhoMhJGWKt5eTJkyQkJNCsWTOn44iIiJQJF7wmy1qbB9wPLCJ/OoavrbXbjDETjTEjL7D7VcAWY8zPwEzgHmtt8sWGzMrKok6dOipYZZQxhjp16uhIo4iISBHFmifLWrsAWHDWY//4jW37Ffn6G+Cby8hXSAWrbNPrIyIi8mua8V1ERESkFKhklWHx8fF8/vnnF73f+PHjmTlz5iV/3+XLl7N27dpL3l9ERERUssq0Sy1ZxZGXl/eb61SyRESkvMvLyyPpWKKjGVSyiunTTz+la9euREREcPfdd+NyuThw4ACtWrXixIkTuN1u+vTpw+LFi4mPj6dt27aMGzeO8PBwxo4dS0ZGBgCxsbH07duXzp07M3jwYBIT838B4uLiGDBgAB07diQyMpK9e/fy+OOPs2rVKiIiIpgyZQoul4tHH32ULl26EB4ezrvvvgvkf7rv/vvvp3379gwbNozjx4+fcwz9+vXjySefpG/fvrzxxhvMnTuXbt260alTJwYMGMCxY8eIj49n2rRpTJkyhYiICFatWkVSUhK/+93v6NKlC126dGHNmjVX5ocuIiJyCRKPHGLLS4NJnj6C3Jxsx3IU68L3suTZudvYfiT1whtehPYNq/P0iNDfXL9jxw6++uor1qxZg7e3N3/84x/57LPPuP3223nssce455576NatG+3bt2fQoEHEx8eza9cu/v3vf9OrVy/uuOMO/vWvf/Hggw/ywAMPMGfOHOrWrctXX33FU089xfvvv8+tt97K448/zpgxY8jKysLtdjN58mReeeUV5s3Ln0h/+vTp1KhRg40bN5KdnU2vXr0YNGgQmzZtYteuXWzdupVjx47Rvn177rjjjnOO5fTp06xYsQKAU6dOsW7dOowxvPfee7z00ku8+uqr3HPPPfj7+/PII48AcMstt/DQQw/Ru3dvDh48yODBg9mxY0eJvgYiIiIlYf2yaEJW/IlQm8bOiCfw9vZxLEu5K1lO+P7774mNjaVLly4AZGZmUq9ePQDuvPNOZsyYwbRp09i8eXPhPo0bN6ZXr14A3HbbbUydOpUhQ4bwyy+/MHDgQABcLhcNGjTgzJkzHD58mDFjxgD5E3uey+LFi9myZUvh9VYpKSns2bOHlStXcvPNN+Pp6UnDhg255pprfnMsN954Y+HXCQkJ3HjjjSQmJpKTk/Obc1wtXbqU7dv/ez/w1NRUzpw5Q0BAwPl/cCIiIldIVnYOqz54gmsS/81RzwacvOErOrbt5mimcleyznfEqbRYaxk3bhwvvPDC/6zLyMggISEBgLS0tMLicfaUBsYYrLWEhoby448//mpdamrxjsxZa3nzzTcZPHjwrx5fsGBBsadQ8PPzK/z6gQce4OGHH2bkyJEsX76cZ5555pz7uN1ufvzxR6pWrVqs7yEiInIl7d2/l9RPf89A189sCxxEqzvew8evhtOxdE1WcfTv35+ZM2cWXuuUnJzMgQMHAHjssce49dZbmThxInfddVfhPgcPHiwsU1988QW9e/emTZs2JCUlFT6em5vLtm3bqF69OsHBwcyePRuA7OxsMjIyCAgI4MyZM4XPOXjwYN555x1yc3MB2L17N+np6Vx11VV8+eWXuFwuEhMTWbZsWbHGlZKSQqNG+ff6/uijjwofP/v7Dho0iLfeeqtwuegROxEREadYa/l+/tdU//Bq2rl2sKvb84Te/3WZKFigklUs7du3Z9KkSQwaNIjw8HAGDhxIYmIiK1asYOPGjYVFy8fHhw8++ACAdu3a8dFHHxEeHk5ycjL33nsvPj4+zJw5k8cee4yOHTsSERFR+Cm+Tz75hKlTpxIeHk7Pnj05evQo4eHheHl50bFjR6ZMmcKdd95J+/btiYyMJCwsjLvvvpu8vDzGjBlDq1at6NChA/feey99+/Yt1rieeeYZrr/+evr06UNgYOGdkRgxYgSzZs0qvPB96tSpxMTEEB4eTvv27Zk2bVrJ/5BFREQuQkp6JgvffICrN0wgx7sG6eMW02bofVCGJsc21v7P/ZgdFRUVZWNifnX/aHbs2EG7du0cSnTx4uPjGT58OL/88ovTUa6o8vY6iYhI+bRl+w5cM+6gk93OzqCRtB7/Dh6+/o5kMcbEWmujzrWu3F2TJSIiIpWTy21Z8O3H9Nz6FFVNLvFXvUbba/7gdKzfpJJVCkJCQirdUSwREZHSdOzUGTb8+yFGpM0goUpzqoz7jJBG7Z2OdV4qWSIiIlKmrY3dhN/cCYxgN3ubXE/z26ZifKo5HeuCVLJERESkTMrJczP7i+kMipuIt3FzdOC/aNHrVqdjFZtKloiIiJQ5+4+dYssHD3JD1hwOV2tD4O8/J6heS6djXRSVLBERESlTFq9eS8Ml9zHK7ONAy/9H05teBa8qTse6aJonqxQsX768cP6rS+Xvf3kfRX399dcLb0otIiJSHqRn5/HRe1PovuQ6QjyOkzz8A5re9la5LFigklUqSqJkXYi1Frfb/ZvrVbJERKQ82XbgGEtfvo1xCc+QXr0lvvevoXbUdU7HuiwqWcU0evRoOnfuTGhoKNOnTy98/LvvviMyMpKOHTvSv39/4uPjmTZtGlOmTCmcMX38+PGFN3WG/x6lSktLo3///kRGRtKhQwfmzJlz3gzx8fG0a9eOP/7xj0RGRnLo0CHuvfdeoqKiCA0N5emnnwZg6tSpHDlyhKuvvpqrr74ayL+5dI8ePYiMjOT6668nLS2tpH9EIiIiF81ay8xFy/B4fyCj8hZyJHQCDf68DK86IU5Hu2zlb8b3hY/D0a0l+02DOsDQyefdJDk5mdq1a5OZmUmXLl1YsWIFbrebyMhIVq5cSbNmzQq3eeaZZ/D39+eRRx4BYPz48QwfPpyxY8cC+SUrLS2NvLw8MjIyqF69OidOnKB79+7s2bMHY0zhNkXFx8fTvHlz1q5dS/fu3X+Vy+Vy0b9//8Jb84SEhBATE0NgYCAnTpzguuuuY+HChfj5+fHiiy+SnZ3NP/7xjxL9MWrGdxERuRgn07L55sPXuDVpCm7PKjD6HQLChzsd66JoxvcSMHXqVGbNmgXAoUOH2LNnD0lJSVx11VU0a9YMgNq1a1/Uc1prefLJJ1m5ciUeHh4cPnyYY8eOERQU9Jv7NG3atLBgAXz99ddMnz6dvLw8EhMT2b59O+Hh4b/aZ926dWzfvp1evXoBkJOTQ48ePS4qq4iISElat+sQx7/8ExPsDxyrHUm98Z9gagQ7HatElb+SdYEjTqVh+fLlLF26lB9//JFq1arRr18/srKysNZiinEjSi8vr8Lrp6y15OTkAPDZZ5+RlJREbGws3t7ehISEkJWVdd7n8vPzK/x6//79vPLKK2zcuJFatWoxfvz4c+5vrWXgwIF88cUXFzNsERGREpfncvNJ9Hf02vQIXT2OkNTpT9Qf/jR4lr9KciG6JqsYUlJSqFWrFtWqVWPnzp2sW7cOgB49erBixQr2798P5J+6AwgICODMmTOF+4eEhBAbGwvAnDlzyM3NLXzeevXq4e3tzbJlyzhw4MBF5UpNTcXPz48aNWpw7NgxFi5cWLiuaIbu3buzZs0a4uLiAMjIyGD37t2X8qMQERG5ZAnJ6bz7xrPctHkcDbwzyLl5BnVH/bNCFiwoj0eyHDBkyBCmTZtGeHg4bdq0KTxdV7duXaZPn851112H2+2mXr16LFmyhBEjRjB27FjmzJnDm2++yV133cWoUaPo2rUr/fv3LzwadeuttzJixAiioqKIiIigbdu2F5WrY8eOdOrUidDQUJo3b154OhBgwoQJDB06lAYNGrBs2TI+/PBDbr75ZrKzswGYNGkSrVu3LqGfkIiIyPkt/mkPOdF/5j5Wk1SvO3Vv/xgC6jsdq1SVvwvfpczS6yQiImfLynUxfcYchu18khCPY5zp9hdqDn4CPDydjlYidOG7iIiIXHG7j6ay8MPnuSfzPXKq1MTeHE3N5n2cjnXFqGSJiIhIibLWMmPNNgIW/4UHPdaR3PAqat/2AfgFOh3tiio3Jau4n+QTZ5S1084iIuKMlIxc3vliBjcfeJpgjxOk9fk7ta9+GDwq32ftykXJ8vX15eTJk9SpU0dFqwyy1nLy5El8fX2djiIiIg6KjT/Jqk8n8XDuR2RXrYu5ZSH+TbtfeMcKqlyUrODgYBISEkhKSnI6ivwGX19fgoMr1iRyIiJSPC635YOlP9F09aP82SOWlKYDqXHTdKh2cZN0VzTlomR5e3sXzqouIiIiZcex1Cz+9fHn3JX0HEEep8nqP4kave8HnXkqHyVLREREyp4fdiSy9etJ/N39OVl+DfG8ZTFewZ2djlVmqGSJiIjIRcnOc/HW3HVE/vQED3r+TFrLYfhf/w741nA6WpmikiUiIiLFtv9EOtM+/piHUl4k0Cud3MGv4N/tTp0ePAeVLBERESmWb2MPkDBnEs97zCAroClet0ZDg3CnY5VZKlkiIiJyXmnZebw8cwUDd/6d6zy3kdn2OvzGTIUqAU5HK9NUskREROQ3bU1I4cNPP+DxzNeo6Z2Fe9ibVI38fzo9WAzFmn7VGDPEGLPLGBNnjHn8PNuNNcZYY0xUkceeKNhvlzFmcEmEFhERkdJlreXfK3az8t0HeTnrGfxq1sP7nhV4dL5dBauYLngkyxjjCbwNDAQSgI3GmGhr7faztgsA/gSsL/JYe+AmIBRoCCw1xrS21rpKbggiIiJSkk6mZTPpi++56dCzdPPcSXaHm6k24lXw8XM6WrlSnNOFXYE4a+0+AGPMl8AoYPtZ2/0TeAl4pMhjo4AvrbXZwH5jTFzB8/14ucFFRESk5K2NO8HXX/ybp/PexN87DztiGlUibnY6VrlUnNOFjYBDRZYTCh4rZIzpBDS21s672H0L9p9gjIkxxsTo1jkiIiJXXq7LzasLf2Hrhw/yuut5qtYJxvveVRgVrEtWnCNZ5zrxagtXGuMBTAHGX+y+hQ9YOx2YDhAVFfU/60VERKT0HErOYNJn33F30nNEesWR22k8vtdOBu+qTkcr14pTshKAxkWWg4EjRZYDgDBgucm/EC4IiDbGjCzGviIiIuKgBVsTWfTNe7xs36GqDzD6A7zDrnM6VoVQnJK1EWhljGkGHCb/QvZb/rPSWpsCBP5n2RizHHjEWhtjjMkEPjfGvEb+he+tgA0lF19EREQuRWaOi+eiN9N880u84fUdOfXD8b7pI6jd3OloFcYFS5a1Ns8Ycz+wCPAE3rfWbjPGTARirLXR59l3mzHma/Ivks8D7tMnC0VERJy1ak8S0+d8z6Opkwn32o+r6z34DJoIXlWcjlahGGvL1iVQUVFRNiYmxukYIiIiFc7epDRenruZFns/4j7vaLx9fPAe8w60G+50tHLLGBNrrY061zrN+C4iIlLBnc7I4Y2lu0jd8DnPeH5FkPdJXG2H4znkBajZxOl4FZZKloiISAWV63Lz2boDrFgazUOuDwn32kdu/Y4w9BM8Q3o5Ha/CU8kSERGpYKy1LN+VxPtzv+eW1H/zgedGcgMawKDpeHe4HjyKdVc9uUwqWSIiIhXI7mNneG3uBqLi/4/3vZZgqvhg+zyFd4/7waea0/EqFZUsERGRCuBkWjZTF2/D66cPmOz1LdW9MrARt+HZ/28QUN/peJWSSpaIiEg5lpPn5uO1+9nywxc8ZD+hmddRcpv2xWPo8xAU5nS8Sk0lS0REpByy1rJ4+zFmzp3HHzLe406PHeTUbgVD38S71UAw57qznVxJKlkiIiLlzPYjqbw1ZyXXHHmXdz1XkVe1Jgx4FZ/I8eCpt/ayQq+EiIhIOXH8TBZvLdxMnS3v8qrXPHy8LPT4Ez5X/QV8azgdT86ikiUiIlLGZeW6+GB1HIeXvc8D5kvqe50mp+1oPAc/C7VCnI4nv0ElS0REpIyy1rJg61EWz/uKu7Pep73HAbLqd4bhk/Fp3NXpeHIBKlkiIiJl0JaE0/x79iJGHHuHNzw3kRUQDEPfxzf0Ol3UXk6oZImIiJQhR1OyeHv+Olpuf4tXvb7HXaUa7r7P4tv9HvD2dTqeXASVLBERkTIgM8fFv5fvJGP1v3jUfIu/VzZ5kePx6f8U+AU6HU8ugUqWiIiIg9xuS/Tmw2xc8AF353xEE48kMkMG4DHseXzqtnE6nlwGlSwRERGHxB44xZezvuXG5GmM9thNRp02MHw6VVtc43Q0KQEqWSIiIlfY4dOZTI9eTuSeqbzsuZasqnVwD5xKtcjbwMPT6XhSQlSyRERErpD07Dze//5nfNa9wZNmAZ7eHuR0/wu+fR+CKgFOx5MSppIlIiJSytxuy7cx8cR99y/udH1JoEcqGe3GUmXIs1Aj2Ol4UkpUskRERErR+n0nmT/rE25Lmc5Yj8OcadAVRrxItUaRTkeTUqaSJSIiUgoOnszgo9nz6Rv/BhM9t5IW0Bg77BMC2o3QZKKVhEqWiIhICTqTlcv7i9dTf+NrPOnxA7lV/Mnp9xz+3SeAl4/T8eQKUskSEREpAS63Zea6PSQteY0/uGdR1TOXrE534jfwSahW2+l44gCVLBERkcu0Zs9xVs+axm3pH9LInCQlZDCeI57HL7Cl09HEQSpZIiIil2hfUhpff/s1Qw6/yWMe+0ip1R476iNqNOvjdDQpA1SyRERELlJKRi4fL1hGqy0v87jHBtJ865Ez+F/U6HQzeHg4HU/KCJUsERGRYsp1uZm5+hdyl73I3XYheHqT3uMx/Pv9GXyqOR1PyhiVLBERkWJYvuMwv8yewq1ZX1DDpJPS9gZqDXsWn+oNnI4mZZRKloiIyHnsOZrK/JnvM+L4NPp5JHKyfg/MmBep1aCj09GkjFPJEhEROYfk9By+ip5Lpx2v8GeP7Zz2b0buiC+p03aIJhOVYlHJEhERKZDncrNh/0l+3riCBrs+4W67gkzv6qT3m0zNnneCp7fTEaUcUckSEZFKLSfPzbq4o+xav4iA+EX0cW+gpzlJrvHmdMTd1B78BFSt6XRMKYdUskREpNLJynWxZsdBDm6YS2DCEvrYWK4y6eQYH0417EN259FUaTeM2n51nI4q5ZhKloiIVAoZOXms+XkXx2Jm0/DoD/TiZ/qbXDI8AjjdeCDVoq7Dp80A6vv4OR1VKgiVLBERqbBSs3JZF/MTp3+aRcjJ5VzDTjyN5bRPPU42u5n63cZSLaQn1XStlZQClSwREalQTqVls3HdSjK2zqHN6ZUMMgcAOFq1BYmt7qdB999Rs2EENfUJQSllKlkiIlLuHU9JY9OaRbi3zyXszGoGmSTcGBICwjnU7jYadRtLUGBzp2NKJVOskmWMGQK8AXgC71lrJ5+1/h7gPsAFpAETrLXbjTEhwA5gV8Gm66y195RMdBERqcyOnEhm+8rZeO5ZQMeMHxls0sjBm4O1u3E47FEadhtDE/96TseUSuyCJcsY4wm8DQwEEoCNxphoa+32Ipt9bq2dVrD9SOA1YEjBur3W2oiSjS0iIpXRoYRD7FnzLVX3fkfH7FgGmGzS8ONQvT5kRoymYedhtPSt7nRMEaB4R7K6AnHW2n0AxpgvgVFAYcmy1qYW2d4PsCUZUkREKq/9cTs5uHYGNQ4uJiz3FxobNyc86rC30UgCo66jQfgA2nn5OB1T5H8Up2Q1Ag4VWU4Aup29kTHmPuBhwAe4psiqZsaYTUAq8Ddr7apz7DsBmADQpEmTYocXEZGKx7rd7N2+kWPrv6Hu4aW0du+lGXDIswm/NPs9DbqNpX6b7gR6eDgdVeS8ilOyzvXxi/85UmWtfRt42xhzC/A3YByQCDSx1p40xnQGZhtjQs868oW1djowHSAqKkpHwUREKhnrymNP7A+c+mkWjY79QEt7lJbAbu92xLb8M0173UDjpqE0djqoyEUoTslKgF/9XgcDR86z/ZfAOwDW2mwgu+DrWGPMXqA1EHNJaUVEpMJw5WQSt34+6Ztn0+zkClqTSo71YkfVTiS2nECL3tfTOkhnN6T8Kk7J2gi0MsY0Aw4DNwG3FN3AGNPKWrunYHEYsKfg8bpAsrXWZYxpDrQC9pVUeBERKV/y0pLZu3YWOduiaZGyjjZkkWarss2/O7QZRts+v6NjrdpOxxQpERcsWdbaPGPM/cAi8qdweN9au80YMxGIsdZGA/cbYwYAucAp8k8VAlwFTDTG5JE/vcM91trk0hiIiIiUTdnJB4lfMwOzcwHN0jfRBhdJtiaxNQbiHTaSsF7D6OanW9lIxWOsLVuXQEVFRdmYGJ1NFBEpt6wl68h2Dq6dQZW9C2matROA/bYBe+v0wz9iNBHd++Pro1vZSPlnjIm11kada51mfBcRkcvndpOxfz2H182gevwi6ucm0BrYSksW1p9A7cgxRHTuRjMvT6eTilwxKlkiInJp8rJJ3/kDxzZ8Q+3D31PTlUyI9STGI4z1wTfSoOt1dAptTwdPTbUglZNKloiIXJS8pL0kzHue+gcX4GczqGd9We8ZyelmgwjpPoaurZri6aGbL4uoZImISLHkHI/jcPSzNE6YR5D15Huvq0hvcS1tegznmpD6GKNiJVKUSpaIiJxX9tFdJERPJOTIAoKsF/OqjqTGgL8wNLIDHjpiJfKbVLJEROScso7sICF6Is2OfkdD68W8aqMJHPwIozq211ErkWJQyRIRkV9JP/wLR+b8kxbHF9HQ+jDf/zoaDH2UkaFtVK5ELoJKloiIAHDm4BYSoyfS8sTS/HJV/QaChz3KyLatnI4mUi6pZImIVHIp8Zs5Gv0sbZJ/wFhfFta8mSbDH2VEq+ZORxMp11SyREQqqeS9sRyfN5G2p5ZjbFUW1LmN5sP/yrDmTZ2OJlIhqGSJiFQyJ/ZsJGneRNqlrMTLVuO7wHG0GvUo1zZp7HQ0kQpFJUtEpJI4uvNHTi34J+1S1+Bt/Vhc7/e0Gf1XhjRq6HQ0kQpJJUtEpII7vG0NKQv/Sfu0H/G1fixpcCftRj3KoAZBTkcTqdBUskREKqiDW1aQtug52qevp5r15/uGEwgd8ygD69VzOppIpaCSJSJSwezb9AOZS54nNGMjp2wAyxr/kbAxf6F/nUCno4lUKipZIiIVRNzGJeR8/zzts34i2Qawoun9dBjzF66uVdvpaCKVkkqWiEg5t2PdQtzLJhOavZmT1GBVswcJH/0wfWvUdDqaSKWmkiUiUg5Za9m2Zj5m5YuE5mzhBDVZ0+JhOo55iD7+1Z2OJyKoZImIlCvW7Wbzqrn4rH6JsNxfSKIW61o/SsdRD9LLL8DpeCJShEqWiEg54Ha5+WnFbKqufYVOedtIojYb2j1O+IgH6F7N3+l4InIOKlkiImWYy+Um5oeZ+K97jSjXDo6bOsSGPkmHEQ/Q1bea0/FE5DxUskREyqC8PBfrl3xNrY2v0c29m2MmkM3hfyds2H3Uq1LV6XgiUgwqWSIiZUhOrot1i74g8KfX6eXewzFTly0RzxB67b3U9/F1Op6IXASVLBGRMiArJ48fF35K0OY3ucrGccyjPts6T6LdkAnU967idDwRuQQqWSIiDsrMzmPN/I8I3vomV9v9HPUIYkeX52k76C7qe/k4HU9ELoNKloiIA9Kyclg990OabXubAcRz1LMBe7q+SMv+dxCkciVSIahkiYhcQSkZ2ayJfp8WO99hCAc46tWIfd1fofnVvyfIU3+SRSoS/YsWEbkCktOyWDPnPdrunsa15hCJ3o050PN1ml71/0DlSqRC0r9sEZFSdDwlnbXR7xEWN50RJoHEKk051PtNGve+FTw8nY4nIqVIJUtEpBQcPZXOmjnvErH//xhtjpBYJYTEvv+iQY+bVK5EKgmVLBGREnI8NZOYH5eR80s0nVJ/4HfmGIm+zTh29bs06HoDeHg4HVFEriCVLBGRy3DkZCpbVs+HXfMJT1/DtSYZFx4crhHBid6TaBA1VuVKpJJSyRIRuUiHEpPYsXoW3nELiMxazxCTQRY+JAT2JDF8NEFRI2niV8fpmCLiMJUsEZFi2Hcgnn2rZxIQ/x0ROZtpbHJJNdU5HNSfzMgxBHUaSksf3bBZRP5LJUtE5BystezZtZUj62ZS+9ASQvN20NxYjnvUY0+T66nX5XfUC+1HdU2/ICK/QX8dREQKWLeb3T+vJSnmW4ISv6e1O57WwAGvZvzS8m6Ce1xPvRadqWeM01FFpBwoVskyxgwB3gA8gfestZPPWn8PcB/gAnQ5i18AACAASURBVNKACdba7QXrngD+ULDuT9baRSUXX0Tk8rjyctm9YTGpm2fR9Pgy2nCCltawxzeMn5o/Skiv62ka3IamTgcVkXLngiXLGOMJvA0MBBKAjcaY6P+UqAKfW2unFWw/EngNGGKMaQ/cBIQCDYGlxpjW1lpXCY9DRKTY8rLS2L02mqyt0TQ/tYp2pJFtvdnhF0VCqwdp3ed62gY2cDqmiJRzxTmS1RWIs9buAzDGfAmMAgpLlrU2tcj2foAt+HoU8KW1NhvYb4yJK3i+H0sgu4hIsWWnJrFvzTe4ts+jxZn1tCeHFOvHzuo98Wg3nHZ9RhMRUNPpmCJSgRSnZDUCDhVZTgC6nb2RMeY+4GHAB7imyL7rztq30SUlFRG5SFkn4tm/6iu89iygWfoW2hk3R20dNtYahm+HUXToOZRuVX2djikiFVRxSta5rvC0//OAtW8DbxtjbgH+Bowr7r7GmAnABIAmTZoUI5KIyDlYS0bCFg6u/Zpq+76jSXYc7YA4GrOi3q1U73Qd4V2u4ipvfeZHREpfcf7SJACNiywHA0fOs/2XwDsXs6+1djowHSAqKup/SpiIyG9yu0jbs4Yj62dS8+Bi6uUl0toatpg2bGvwR+pGXUfHiM609NSs6yJyZRWnZG0EWhljmgGHyb+Q/ZaiGxhjWllr9xQsDgP+83U08Lkx5jXyL3xvBWwoieAiUonlZpK6fQlJG7+l7pEfqO5Ooan1IsYznPVNbie4++8Ib9uaCA9NtSAizrlgybLW5hlj7gcWkT+Fw/vW2m3GmIlAjLU2GrjfGDMAyAVOkX+qkILtvib/Ivk84D59slBELknmKVJ+ns/pTbOof3w11W0W2Kqs94oireUQWvQYTc/mjTCaw0pEyghjbdk6OxcVFWVjYmKcjiEiZUHKYU5tmk3Gz7OpfyoWL1wcszVZ79ODnFZDad/jWtoFB6pYiYhjjDGx1tqoc63T1Z8iUnZYC0k7SY79ltzt86h/Zju1gJPuhnxTbQymzXAie17DyPo1nE4qInJBKlki4iy3G5uwgVOxszC7FlAr6yC1gU3ulnwX8HuqhA6nR7ce3FjHz+mkIiIXRSVLRK48Vy527zJOxX6Lz75F+Ocm4289+dGGsrvmKPw7jqBv53DG1azqdFIRkUumkiUiV05eDlmxn5K97BVqZB3Gx/qywh3B/rr9COw0gmsiWtI3QJODikjFoJIlIqUvL4eMDR+Tt+IVqmcnssvdnB/q/oPGXUZzTYfGDPPzcTqhiEiJU8kSkdKTl036ug9xrXyV6jnH2ORuycqGz9Pv2lt4qEktp9OJiJQqlSwRKXm5WZz58X3sqilUzz1OrLsVPwY/zDXDbubBRvpkoIhUDipZIlJycrNIWfN/mDWvUz33BDHu1mxs+lcGDr+R++tXdzqdiMgVpZIlIpcvN5NTq6bjufYNauSdZIO7LZubP8WgYTdwb11/p9OJiDhCJUtELl1OBidXvIP3+reolZfMenc7fmn1NIOHX0/XWtWcTici4iiVLBG5eDnpJC37F1U2vE0d16n8+a3a/JPBw8bSrYamYBARAZUsEbkY2Wkc/f4tqsW8Q133adbaDuwNfYEhQ6+jR0AVp9OJiJQpKlkicmHZZziyeCoBm94lyJ3CGjpyMOwVhgwdRU/NcSUick4qWSLy27JSObTodWr+PJ2G7jOsJoIjHR9k8JAR9Krq7XQ6EZEyTSVLRP6HzTzNwYVTqLP1PRrbNFaZSJI6/5nBg4bRu4r+bIiIFIf+WopIIZt5ivj5r1F32/s0tWmsNFGc6voQgwYMpaqPp9PxRETKFZUsEcGdfop9814maOeHNLPprPToSlr3h+nffxBVvFSuREQuhUqWSCXmSk9mb/SLNNr1MS3JYKVnd7J6PsLV/frj7enhdDwRkXJNJUukEso7k0TcnBdpEvcJrclipVdP8no/ylV9+uGlciUiUiJUskQqkZyU48TNeYGQfZ/T2mazyqc3pu+j9O55FR4exul4IiIVikqWSCWQdfoocbNfoHn8l7S12ayu0gfvqx/jqu69MEblSkSkNKhkiVRgGclHiJv9PK0Ofk07m8Paqn3x7f84faK6q1yJiJQylSyRCijtxCHiZr1Am8MzCLW5rK12Df4Dn6B3pyiVKxGRK0QlS6QCST12kL2zJ9Eu8VvCrIt1/v2pOfhx+oRHOR1NRKTSUckSqQBOJcazb/Ykwo7OpgMu1gUMInDoE/QOjXA6mohIpaWSJVKOnTi8l/g5kwg/Fk04lg01BlN/2JP0btPB6WgiIpWeSpZIOXTs4B4ORk+iY9JcOgIxtYbSYPhT9GrZ3uloIiJSQCVLpBw5Er+LQ9GT6HRyPrWA2DrDCR7+FD2at3E6moiInEUlS6QcOBi3jcR5zxF56jsCMfxUdyRNRz5FjyatnI4mIiK/QSVLpAzbt2srx+Y/R5eURdTHk031xtBs9FN0b9Tc6WgiInIBKlkiZYy1lh1bNpKy9BW6pC6hIZ5sDhpL8zF/o2tQU6fjiYhIMalkiZQBbpeLnT+t4GTsLBof+5729jCZ+PBzwxtpMfpJouo3cTqiiIhcJJUsEYfk5WSxa91C0n6eTbOTK2lPMnnWg91VO7Kp5ThaXn0bnes0cjqmiIhcIpUskSsoJz2F3Wtmk7stmpYpawklgwxbhV3+XUloO4zWfcbSvmZdp2OKiEgJUMkSKWVZpxKJWz0Ts2s+Lc/EEGZySbYBbKvZD+/QEbTvNZJOfv5OxxQRkRKmkiVSCtITdxO/5mt84xbSLHMbYcZymLqsDxyDf8dRhHYfRHcfH6djiohIKVLJEikJ1nImPoaEtTMIiF9McO5+QoFdNGNZ0O+p1fl3dIjsQSMvT6eTiojIFVKskmWMGQK8AXgC71lrJ5+1/mHgTiAPSALusNYeKFjnArYWbHrQWjuyhLKLOMuVS8quFRxbP5M6CUup40qitTVs9ghla/CDBHX9HeFh4bTxME4nFRERB1ywZBljPIG3gYFAArDRGBNtrd1eZLNNQJS1NsMYcy/wEnBjwbpMa21ECecWcUZOOqe3fsfJ2G+pn7icGjaNKtabjV6dSGl2N016XEenls3xULESEan0inMkqysQZ63dB2CM+RIYBRSWLGvtsiLbrwNuK8mQIo5KP0HypmjObJ5N0IkfqUkOWD/W+nQls/lQWvUcSe8mQRijYiUiIv9VnJLVCDhUZDkB6Hae7f8ALCyy7GuMiSH/VOJka+3si04pcqWdiudk7Gyyts4hKGUztXGTYQNZ6DsYd+thdOg5hEENajmdUkREyrDilKxz/ee5PeeGxtwGRAF9izzcxFp7xBjTHPjBGLPVWrv3rP0mABMAmjTRzNbiAGuxR7dwMmYWduc86qbvoQ6ww92Y1f434tl+OFHd+jG6rqZaEBGR4ilOyUoAGhdZDgaOnL2RMWYA8BTQ11qb/Z/HrbVHCv5/nzFmOdAJ+FXJstZOB6YDREVFnbPAiZQ4Vx724I+cjJ2F154F1MxOpLY1xNrWLKhxJ1U7jKJX1y7cWLOq00lFRKQcKk7J2gi0MsY0Aw4DNwG3FN3AGNMJeBcYYq09XuTxWkCGtTbbGBMI9CL/ongRZ+Rm4o77geTYb6kWv4RqeSkEWG9W2w7srXMTtSJG0jeyPV0CfJ1OKiIi5dwFS5a1Ns8Ycz+wiPwpHN631m4zxkwEYqy10cDLgD8wo+Di3/9M1dAOeNcY4wY8yL8ma/s5v5FIaclIxr3rO079NIuAwyvxcWfhY6uxxHYiod41NOg8nH7hzenvp8lBRUSk5Bhry9bZuaioKBsTE+N0DCnvTh/CtWM+ZzbPIuDYRjxxcdTW4nvbheMN+9OsyyCubt+YGlW9nU4qIiLlmDEm1lobda51mvFdKgZr4fgOcrdFk7k1muqntuEJJLkb8ZUZSUrTQYR27svotvXxq6JfexERKX16t5Hyy+2CQxvI3T6XnF/m4pd+EG9gi7sVKzxuJbvFEKI6d2Ncq0B8vXU7GxERubJUsqT8ST9BzpKJuHfMwzf7JNZ6stEdxhqvIdBmKL06hXF/i0B8vDycTioiIpWYSpaUL4lbSP/4BrwyTrDE3ZkffcbhFzqEqzu24LGQ2nh5qliJiEjZoJIl5YZry0zcs/5IituPt+u9xnXDhzOpcS3dJ1BERMoklSwp+9wusr57Gt8Nb/KTuzUrI6bw7KieOmolIiJlmkqWlG2Zp0n7fDz+h5bxhWsA3iNe5i9dmzudSkRE5IJUsqTsStpF+kfXU+VMAi943s3gO54gsoluyiwiIuWDSpaUSe4dC8ideScZeZ5MqjWZP99xO/Wr61Y3IiJSfqhkSdnidpO9/CW8V05mlzuEOW1e4ukb+mueKxERKXdUsqTsyE4j/esJ+O2dzyxXb1IHvMLfrmpLwf0wRUREyhWVLCkbkveT/vGN+J7ezavmdrrf/g/GtKrrdCoREZFLppIljrN7l5H9xThyc/OY5Pcs9/7hLprUqeZ0LBERkcuikiXOsZbcNW/hufQfHHA35JOQF/jbrdfqBs4iIlIh6N1MnJGbSea3f6Lqjq/5ztWF/b1fYeLAjpq9XUREKgyVLLnyUg6T/slN+J3Ywlvu62l140TuDWvodCoREZESpZIlV9bBdWR+dgs2K50nfZ9g/B330bp+gNOpRERESpxKllwxeRs/wCx4hKOuOkxr+CZP3D6amtV8nI4lIiJSKlSypPTl5ZA176/4bv6AFa5wNnZ+medGdNUNnkVEpEJTyZLSlZZE+qe34Hd0A9PdIwgc/RyPdG7qdCoREZFSp5IlpefIJjI/uRnPjBP83evPjP3DQ3RsXNPpVCIiIleESpaUCvfPX+Oacz8nXQG8XvtV/vr7G6mnGzyLiEglopIlJcvtIue7v+Oz4W02utuyqP1LPDe2N1W8dINnERGpXFSypORkniLj83FUO7SCj12DYPDz/L1XS93gWUREKiWVLCkZx3eQ+fENeKcdZqK5hwHjH6Vni0CnU4mIiDhGJUsum90xl9yZE0jL8+b5gBd4+I7/R+PausGziIhUbipZcuncbnKXTcZ71YtsdzfnqxYv8NzNA6jmo18rERERvRvKpck+Q+bXd1F170Jmuq4iqe8LPN8/VNdfiYiIFFDJkot3ci+Zn9yE9+k4JtvxRN30BGNDg5xOJSIiUqaoZMnFifuenC/HkZVrearq0/zxjj/Qsp5u8CwiInI2lSwpHmtxrZmKWfoMe92NeC/4OZ6+7VpqVPN2OpmIiEiZpJIlF5abSfa391FlxzfMd3VlR9fJvDQsEk8PXX8lIiLyW1Sy5PxOHyLz05uocmIbU1w30mzMP3gkMtjpVCIiImWeSpb8tgNryf7sVlzZmfzV+wnG3XkPHYJrOJ1KRESkXFDJknNyb/g3duGjJLjq8Ubdyfx9/GjqBlRxOpaIiEi5oZIlv5aXQ868v+Cz+WN+cEWwKvx5XhnTAx8vD6eTiYiIlCsqWfJfZ46R+fmtVE3cyDuuUQRc+wz/6N5ME4yKiIhcgmIdnjDGDDHG7DLGxBljHj/H+oeNMduNMVuMMd8bY5oWWTfOGLOn4H/jSjK8lKDDP5H1rz7YxC08Zh4m8vdTuK1HcxUsERGRS3TBI1nGGE/gbWAgkABsNMZEW2u3F9lsExBlrc0wxtwLvATcaIypDTwNRAEWiC3Y91RJD0Qund38Ba45f+KEuzovVH+ZJ+64nuBausGziIjI5SjOkayuQJy1dp+1Ngf4EhhVdANr7TJrbUbB4jrgP5/xHwwssdYmFxSrJcCQkokul82VR96CxzGz72FDXkvebPkerzxwmwqWiIhICSjONVmNgENFlhOAbufZ/g/AwvPs2+jsHYwxE4AJAE2aNClGJLlsGclkfTkO34Mr+SBvMNnXPMvkq9vq9KCIiEgJKU7JOte7rj3nhsbcRv6pwb4Xs6+1djowHSAqKuqczy0l6Ng2sj65EY+0RP5u7+Xq2x7imrb1nU4lIiJSoRSnZCUAjYssBwNHzt7IGDMAeAroa63NLrJvv7P2XX4pQaWEbJ9D7jd3k5JXhYnVnueh399Ky3r+TqcSERGpcIpzTdZGoJUxppkxxge4CYguuoExphPwLjDSWnu8yKpFwCBjTC1jTC1gUMFjcqW53biWToKvb2drbiMmB7/D83+6QwVLRESklFzwSJa1Ns8Ycz/55cgTeN9au80YMxGIsdZGAy8D/sCMgmt6DlprR1prk40x/yS/qAFMtNYml8pI5LdlpZI94y6q7P2Or/L6cajnP3llSAfd4FlERKQUGWvL1iVQUVFRNiYmxukYFceJOLI+vRGv0/t4wX074WMeZVQn3eBZRESkJBhjYq21UedapxnfK7I9S8n9ajwZufB3n2e4d/x4whrpBs8iIiJXgkpWRWQt7tWvY75/lt3uJrxdfyITxw0l0F83eBYREblSVLIqCmvh2C+wcz552+bglbSdua7u/NRpEq+P7KwbPIuIiFxhKlnlmdsFB9fBznm4ts/FM/UQbgyb3K2Z7b6T9sMf4OnuIU6nFBERqZRUssqb3EzYuwx2zse1awGemcnk4M0qVxiL3EOIq9mbHuFtGR/RiFb1A5xOKyIiUmmpZJUHGcmwZzHsmIs77ns88jJJM34syYtgsSuKo3V70q9DC+7sEESrev66NY6IiEgZoJJVVqUkwM75sHMeNn4Nxro4YWqzILc3i91RZDboxoAOTXgsLIiQQD+n04qIiMhZVLLKCmvh+A7YOR+7cx4mcTMA8R6NmZc7nCXuKHyCIxncoSGTw4IIrlXN4cAiIiJyPipZTnK74NCG/KNVO+djTu0HYJtHG+bm3sxSG0X9ZmEMDQvi/0KDqFfd1+HAIiIiUlwqWVdabhbsX5FfrHYtxKQn4TJebKAD0bl/YAWdad2yFUPDgpjRPojafj5OJxYREZFLoJJ1JWSehj1L8otV3FJMThpZHtVYYTsxN+cm1nh0Iqp1U4aGBfF4u/rUqOrtdGIRERG5TCpZpSX1SMGF6/Ox8asw7jxSvWqzKK8H83Ii2ezVgd5tGzE0LIgX29TDr4peChERkYpE7+wlxVo4sRt2zssvV4djATjuHcw817XMzenMXtrQv10DbgkL4t3WdfH19nQ4tIiIiJQWlazL4Xbnl6mdc/OL1ck4APZXacss903Mz+1MsmnKoA4N+FOHIHq1CNTtbURERCoJlayLlZcN+1flH7HatQDSjuE2Xuyo0pEZrjtYmBuJ27sBgyPr88+wBnRtVhsvTxUrERGRykYlqziyUiFuCeyYl38Be84Zcj2r8pNPF77Mu57v8zri71OHIV0b8FaHIDo3qYWHh2ZdFxERqcxUsn7LmaP5R6p2zod9K8CdS6Z3LdZ69eLz3A6szgolyLcmQ3s14JOwIMKDa+h2NiIiIlJIJauoE3H/vXA9YSNgSfUNZlmVEXx8ugObslrRol51hvYN4i9hDWjXIEDFSkRERM6pcpcstxsSN+WfBtw5H07sAuC4f1sWVb2VT053YHdWMKENazC0SxAvhjWgZT1/h0OLiIhIeVD5SpbbVTDj+nzYuQDOHMEaTw5V78Rc3wl8djqMI1mBRDSuye+6BzE0rAFN6ug+gSIiInJxKl/Jsm7sjPHY3Gz2VO/ON1Vu4KuU9qRm+dMlpDZ39QlicGgQDWtWdTqpiIiIlGOVrmRluT24n3+wOr02uZm+9GxRh79eE8Sg9kHUDajidDwRERGpICpdyfL19qRlx94MquvHwHb1qaUbMIuIiEgpqHQlC+DxoW2djiAiIiIVnKYiFxERESkFKlkiIiIipUAlS0RERKQUqGSJiIiIlAKVLBEREZFSoJIlIiIiUgpUskRERERKgUqWiIiISClQyRIREREpBSpZIiIiIqXAWGudzvArxpgk4MAV+FaBwIkr8H2cUtHHBxV/jBpf+VfRx6jxlX8VfYxXYnxNrbV1z7WizJWsK8UYE2OtjXI6R2mp6OODij9Gja/8q+hj1PjKv4o+RqfHp9OFIiIiIqVAJUtERESkFFTmkjXd6QClrKKPDyr+GDW+8q+ij1HjK/8q+hgdHV+lvSZLREREpDRV5iNZIiIiIqVGJUtERESkFKhkiYiIiJQClayLYIypaYzxcjpHaTPGVNjfC2NMjQo+vsCK/jtqjGlqjKld8LVxOk9JM8aEGWMaOp2jtBhjwo0xA40x3k5nKS3GmHbGmAeNMYFOZykNxpjGxpgAp3OUlpJ8r6+wbzYlyRjja4z5DJgJvGmMCXI6U0kyxvgbY94xxjzpdJbSUjDGd4EvyH8N6zmdqSQVjO8tYAYwxRjTwulMpcEYcw2wGfgrgK1An9wpeA1fAuYAnZzOU5JMPm9jzIvAPKAx4ONwrBJX8Bq+DKwg/3f0pMORSpQxppoxZjowG5hqjBnudKaSZIypaoz5APgUmGSMOecs7hdDJat4xgJ5wDDAE3jEGHO1s5FKRsERgc+BFsCtxpi21lq3McbT4WglxhjTEvgByABuAhoBfzDGeFaEIyHGmGbARiAXGAq0JH+cFeZIT5FxnAYWA22NMX3PWlduGWMaAbuBqkCYtXZ+kXXlfnwFZbgx0BzoYK1931qbDhVjfADGmAhgE+AC6gMHgQEF68r9GAveEx4DsoGuQDLQuWBdRRifB/nFOBUYXfDwI8aYnpfzvCpZxdMf2G6tzQb+CRwCxhhjqjkb6/JZa5OB18l/c/6c/PFhrXU5mauEnQCesdY+ZK1NBV4EbrfWuirCkRBr7X5gVMH4soC15P8hrDBHeoqMox35r+dXwISz1pVb1trDwDbgZ2ttpjGmjzGmuzGmakUYX4HewF5rbYoxZrgx5j5jTCugopzejgcGWmsfL3jNVgOtoML8jrqADuS/F7qAY4BfRXgfBLDWusk/grzWWpsHvAv0BcYaY6pe6vOqZJ1HkWt3vgU6G2OqWGsPAesAN3CtY+FKQJH/+lhW8I/mPaCxMWZYwfqK8scvDVgKUHAdSCLwS8Gh4XL/X2AA1trdBUfmPgT+DHQzxrxsjOnicLQSUeR1OgDsAtYAOcaYh4wxA51LdvmKHDV+GHjWGBMNvAA8BbxtjOnhWLgS8P/bO9sQO6ozAD/vJpG19YM0klYxQUIabYU1BLuKbNMSbKNCY5Sg0DbYgkbFWH9IggG1sBWrhSqiNn8KNpS2tgXFVs0PxX74I61Noam2BsQYP6LW0jSRQKzRff3xnmumS3b37r13Zs7H+8BlZ+bOvXuec87MnHs+K2n3PLBWRDYDG4EzgXHgprbCNkhU9YCq7q34nhBe1TROnZ8CV4vIn4AbgdOArcAVrYZqAIR0exq4MDz7D2EF52FgVa/f64WsQOfCqD50Q8kW4GXgHY5mpN1YVWkyJfgp/HTS339hpfctYf+DlAohx3IE81DV98P2EezX2ISqHk7pF+ZUfp1joaB8r6ouANZjzaOrUupgPE0adtJpFNivqq8Bp2IP6SXH+kyMTHEdfigiQ6r6PDY79U5VHcPS8HXgglR+8Ex3nwF2AU8B61T1IlX9Dna/GZOE+rlOdx12Tgl/nwauhLRaBqbzU9VHgauBl1R1kap+E/gd8EURObnZkPbGDPeYx7Hn+iPA77G+dYew7kI94YUsIFzgO0RkwRQP3VeAPwOrRWSJqh7E2t3PbjKcvdKFX+c8AR4G3hSRa0MNwVhT4eyHbh0DX8EuJkTkUhH5XO0B7JOZ/CrH/h7238N+CCwMBcvomc6xckPcDWwQkT1Y34lfASdB/E0yXebRcVUdB6sZAQ4Cp4Xmi6jpIo9OAI8CZ4vISDj8LvA28O/mQto73aRh5cf5bqzG/MzGAtgnXebRN4AFYv0IwfLonPBcjJou8uhe4FtYRcNKVd2KDdA4q9f/6YUsQFXfxi70LVO8fxjrbLsXeEhEVmNNhf9oKoz9MJNf5TwNrjuxKuDvYZ03o6cbx0ptwBBwjog8AVyPdaaOmtmkIYCInId13vxn/aEbDNM5Vm6IH2AFyctV9Qqs+WKhiAw3FtAemcFvIvz9+MYvIqPAGqyZLXq6yaOq+gxwJ/B9EVmD9Y98v5kQ9k+312Hgf8Dngf/WGqgB0qXfMPASNvruXKzpd38KNcld5tEjqvqiqv5HRFYCK7Daup7/aZEvrORd3V+M1ViNzPC5G4BtwIa2HWryuwa7qX+9bYcaHd8CXsQ6i7fuMUg/bFTTfdg0B99o22HQjoT1Viv7w2071JCGJwD3hjSM+jrs4xr8GjbgZn3bDnU5hnMfAUYm59uYXj3m0aXAj4DtwFVtO9SRflglw65+82jxC0SLyDrgL6r6qojcCoyq6ppjnDekR6uBZzweC7PwE1VVEfm0Wt+szvE5Gnl/gm4dw7nHAWNqv6g7x6J2nKXfXGC5qu6sHIs6j8LsHMP5/+cUu2MPfstV9W+VfdGIb9azvc8c43jU6Qezvg6j95lMD3l0nla6IsTu3MOzcJHaQLf+aLuU2WBp9qvAcZP2X8BG1G0Fvh2OvwCsDdtSOX9o0v6cusPcst/ctp0acJzXtpOnYX+Osb1yT8Ma/IbadmrAMfdnRe5+fV2DrUdIAxHeqa17DrgjbB+PjUo6A5iPNR3dE95bj80zNLfz+WomApa17VSz32fbdmoiDYnoQZ27X12ObTuVlIa5p18Jju43+2fhIK7B1iOmiUgP2+dhfY1OD/u/xIbY7gCuC8eGwt9dwMZJ3zUfuD+cf1LbbiX4leCYu18Jju6Xtl8Jju7Xnl/rkdNQAmzCqgn3AA+HY3cCv+lEIrZczjVh+3QqVYTALViJd3XbLiX6leCYu18Jju6Xtl8Jju7XvF/rkTLgCB5iUhs/NgR6BzZx4Rg2nHYUmy7/x9gonrXYLO4/wSYiq5aKLwZuI4L+O7n7leCYu18Jju6Xtl8Jju4Xj1/rmX2AkV6NrIWV7Q3A7ZX9EwOnlgAAAuFJREFUm4Fnw/ZS4LvAL4CL2nYo2a8Ex9z9SnB0v7T9SnB0v7j8Wo+wAUf4MPBDrJ31IWA1Vrp9rHLOKdiEhtdP8X1RjXbJ3a8Ex9z9SnB0v7T9SnB0vzj9kp/xXUNsBa7D2ljPxyZJuw94FjhRRDaLyGLgy8CT2KiDj5GwGLRGNs9H7n6Qv2PufpC/o/ul7Qf5O7pfnH7JTUbambq/E+EicglHI3oVsFdVfxbe+zk2ZHMbcC1wIbAPuFFV9zUf+pnJ3Q/yd8zdD/J3dL+0/SB/R/dLxK+J6rJBvahU72Gd1kawNYXuxhaK3QPcVDnnC9iq7/PC/hmV94T4qkOz9ivBMXe/EhzdL22/EhzdLx2/zoK5SaCqE6GqbxwbNfAGsElVd4rIpcAFwHoReQf4AzbZ2B81TP2vtsJ2dRmVqKrxcveD/B1z94P8Hd0vbT/I39H90vFLqk+WiHwJ+DVwCJssbDmwTGw9uu3A48AR4FPAb7HOcQ9M/h6NdJ263P0gf8fc/SB/R/dL2w/yd3S/hPzarBKc7Qu4DJgAFof9LcBdhKVggJXAm8Angc9UPhfN8hQl+5XgmLtfCY7ul7ZfCY7ul45f6wHoIfIfA34QthdhwzevAj4Rjo1WI5zI2ppL9yvBMXe/EhzdL22/EhzdLw2/pJoLA7cDF4vIMlV9Hfgr1j57MoCqPtc5UY2ohqF2Qe5+kL9j7n6Qv6P7BRL1g/wd3S8Qs19yUzgAiMgdwApVvURE5gAnquqBtsM1KHL3g/wdc/eD/B3dL31yd3S/+EmxJgvgQWC/iMzHCrEHOnNqZELufpC/Y+5+kL+j+6VP7o7uFzlJ1mQ5juM4juPETqo1WcDR6fFzJXc/yN8xdz/I39H90id3R/eLF6/JchzHcRzHqYFkS4eO4ziO4zgx44Usx3Ecx3GcGvBCluM4juM4Tg14IctxHMdxHKcGvJDlOI7jOI5TA17IchzHcRzHqYGPAJw1eu+udPSRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(10,5))\n",
    "plt.plot(expected_interaction_rates, label=\"expected rate\")\n",
    "plt.xticks(range(10), labels=new_model_names, rotation=30)\n",
    "plt.plot(estimates_weighting, label=\"actual rate\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "We saw that in case of both of the estimators, we were able to estimate the right click rate for each model. In other words, had we not known the actual order of performance of each model, this technique would help us pick the right model just from the predicted outcomes of each model on the logged entries. It is important to note that in this simulated example, we were able to retrieve the exact click rates. In practice, however, there could be various sources of noise and we may not get the exact numbers back. However, we could aim for three properties, in order of increasing importance, when evaluating offline evaluation methodologies:\n",
    "1. Directionality:\n",
    "    - if online metric for model A > model B, then the counterfactual metric for model A > model B. \n",
    "2. Rate of Change:\n",
    "   - if online metric for model A is 10% more than model B, then the counterfactual metric is also higher by similar amount\n",
    "3. Exact Values:\n",
    "    - the online and counterfactual metrics are very close in absolute value, like our example (this is ideal case)\n",
    "\n",
    "Did you find this helpful? Can you think of scenarios in your daily job where this technique could be useful? Do you see any limitations of this approach not highlighted here? Feel free to drop comments with feedback/criticism/questions; I would love to discuss more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# References / Additional Reading\n",
    "\n",
    "This article is heavily inflenced by [1] and [2]. If you are interested in digging further, [3] [4] [5] are interesting reads with more applications of similar context\n",
    "\n",
    "1. [Counterfactual Reasoning & Learning Systems](https://arxiv.org/abs/1209.2355)\n",
    "2. [Counterfactual Estimation and Optimization of Click Metrics for Search Engines](https://arxiv.org/abs/1403.1891)\n",
    "3. [The Self-Normalized Estimator for Counterfactual Learning](https://papers.nips.cc/paper/5748-the-self-normalized-estimator-for-counterfactual-learning)\n",
    "4. [Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms](https://arxiv.org/abs/1003.5956)\n",
    "5. [Off-policy evaluation for slate recommendation](https://arxiv.org/abs/1605.04812)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
